Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Solaris vs. Linux: Framework for the Comparison

by Dr Nikolai Bezroukov

9. Conclusions

Previous version of conclusions begin to resemble an separate article and was converted into Summing up section ;-).  Business decisions are always compromise and much depends on the goals. The key in OS area is the total cost of ownership.

Here we just reiterate the major points which might help to determine right compromise. Please note that I am talking not about cash-strapped universities and/or start-ups and not about firms located in developing countries. I am talking about making decisions in the environment of more or less well to do (although now far from flush with money) large enterprise IT:

1. Enterprise OS mix behaves like an ecosystem and side effects as well as complexity of the task of adding yet another flavor on Unix in a large enterprise environment (be it linux or Solaris) should not be underestimated. Those side effects tend to eat into savings. Proliferation of unix flavors increase sysadmin overload and lessen the quality of each individual environment as workforce became spread too thin and lack critical mass necessary for acquiring and improving knowledge. With large overload the situation is more about survival then about quality. Even with huge cost-effectiveness of Intel Duo CPUs killing one of existing flavors of Unix usually can save more than adopting a new one.  Cost and tradeoffs typical for excessive diversity of Unix environment are often ignored in simplistic calculations of benefits of linux adoption (or any other new flavor if unix adoption, including Solaris adoption). 
    
2. Enterprise linux flavors are in all dimensions more similar to proprietary software that many people wish to admit and generally suffer from the same weaknesses with some (instability) even more pronounced.   Promises have been made. Assurances have been given. Commitments have been published. But far less has been delivered. It is clear that in some more important in my view aspects linux in technically inferior to Solaris 10, while in others less important from my view aspects Solaris 10 is inferior to enterprise linux flavors.   Stability of enterprise linux distributions is less then stability of Solaris 10 and other major commercial Unix flavors (AIX 5.3 and HP-UX 11i).  Linux distributions are bloated and complexity distributions in general,  and kernel in particular, is high, regression testing is limited and sometimes errors including kernel errors can be introduced during regular patching process due to the update of the kernel or particular important subsystem.  "Blue screen of death" due to driver problems are not uncommon with the only differences from Windows that at the time of the crash linux kernel often is unable to display even the  panic message.  Server simply froze.  Patching in linux is more dangerous process then in Solaris 10 and for critical servers, whenever possible, should probably be limited to minimum subset of security patches.  Due to stability problems linux should preferably be used in applications were redundancy is built-in in the design.  Integration with directories (including Active Directory) is important to the extent that it can be said "It's the Directory, Stupid" and here the favorite is unclear but it might be Suse.  In this area Novell looks slightly better then others, but concerns over cannibalizing eDirectory keep Novell from doing a better job.
    
3. Qualification of sysadmins is the key to stable and reliable work of any Unix or linux server farm. There is no substitutes and neither Solaris not linux can fly well and do not crash without experienced pilots.  High qualification can be achieved only if sysadmin is responsible for no more then two flavors of Unix. That means that the level of diversity of Unix flavors in a large enterprise environment should be tightly controlled.
    
4. We need to distinguish benefits of Intel/AMD hardware from benefits of new OS adoption.   Advantages of linux are too often uncritically mixed with the advantages of switching to Intel/AMD hardware and first of all its dramatically better (often twice or more lower)  price/performance ratio of Intel Duo CPUs in comparison with Itanium and RISK CPUs. In reality, for large enterprise environment with multiple Unix flavors installed, linux offers very limited advantages and some noticeable disadvantages if compared with using existing commercial Unixes on competitive hardware (for example with Solaris 10 on Intel/AMD hardware).  Fashion based replacements of Solaris SPARC servers with linux servers without considering possibility of moving to Solaris 10 on  Intel/AMD hardware increase costs instead of providing savings: the key advantage is faster hardware and Solaris on Intel/AMD similarly beats equal or more expensive Sparc systems. At the same time proliferation of various flavors of Unixes makes infrastructure less cost efficient, not more cost efficient.  That suggests that days of Sparc on low and mid-range servers might be numbered despite T1/T2 CPUs. The latter just slow down the process. Moreover for enterprise class-applications licensing costs and maintenance costs (which include cost of local manpower) by-and-large determine the total cost of ownership.  This situation, for example, is typical for SAP/R3 deployments. higher uniformity cuts costs even if different OSes are used. 
    
5. Linux plays an tremendously progressive role at large enterprise environment as a counterbalance to strangulating IT bureaucracy.  Born as an alternative OS it still can live to its promise. It permits running small, "guerilla" projects and experiment with new technologies like scripting languages.  It also can lessen the negative effects of "pseudo-security" efforts of  "overzealous know-nothings" at desktop area.  Actually, in security area large enterprises IT should fear more from the bungling of the incompetent than from the machinations of the wicked.  In principle, Solaris 10 can play the same role, but it requires more efforts both in installation on corporate desktops and configuring the necessary software.  That might soon change.
    
6. Bureaucratization of IT has very positive influence on linux/Unix adoption and significantly diminishes attractiveness of Windows. The litmus test of the level of bureaucratization is prevalence of form over substance and as a side effect fashion rules and logic does not necessarily prevails in discussions about the relative benefits of introduction of a new OS. IT bureaucracy like any other bureaucracy with age develops goals strictly related to self-preservation.  The more dominant are those self-preservation tendencies the more bizarre and damaging from the point of view of common sense enterprise IT moves can be expected,  the more politically motivated major technological decisions become( misdirected SOX compliance efforts are a good example here ) and the less they care about you, the Unix administrator.  On the other hand the same bureaucracy in Windows space push users to the "Unixland" as bizarre and arbitrary limitations make it difficult to use Windows productively.  With group policies available, Windows world more and more reminds mainframe world.  In this sense linux (and to lesser extent Solaris) serve a very positive and extremely important role in modern IT: the role of "freedom fighters weapon of last resort". 
    
7. The complexity of modern OSes had risen to the level when it is almost beyond the capability of single, even very intelligent, person to understand them. Also OS themselves represent a moving target (linux to more extent then Solaris or, AIX, or HP-UX). Due to this top level admin skills can be acquired only after many years of hard work (forget about people claiming to be "experienced system administrators" with just one or two years of administrator work under the belt unless they are former programmers on the same OS).  Due to the level of variety between different Unix flavors sysadmin skills are to considerable extent Unix flavor specific and that's why usually people tend "naturally" concentrate on a single ("loved") Unix flavor and dislike others (in addition one "minor" flavor can be learned reasonably well too).   Administrators with deep knowledge and passion for Unix flavors currently used represent important part of the company intellectual capital, the capital that can be easily wasted in case of transition.  That actually might can help to explain such a persistent phenomenon as "OS nationalism" often demonstrated in discussions like Solaris vs. Linux as they usually pretty well resemble the style of USA culture vs. Great Britain culture (Canada or Australia can be substituted for Great Britain) discussions (you know both countries share the same language don't they ;-).  Unix sysadmins who moved to a different flavor of Unix much like expatriates often do not feel well in a new territory for several years as considerable part of their skills is Unix flavor specific and the higher qualification they have that large is this "specialized" part.   For administrators with almost a decade of experience in a particular Unix flavor under the belt, to quote  Linux Torvalds, switching from administering one OS to another is not unlike “performing brain surgery on yourself”. This is one of the major reasons why adding any new Unix flavor to the large enterprise Unix mix usually does not provide for expected savings. 
    
8. Due to the complexity (should I say overcomplexity ?)  of modern Unixes the value of OS certification cannot be overestimated. Having certified in particular flavor of Unix administration for at least some administrators on the floor is probably the most reliable way to  avoid rather painful errors and horror stories at the initial stages of introduction of any new flavor of Unix. For example, it is clear from the content of the paper, that the expertise in Solaris or AIX administration is not directly translatable into linux domain and attending one intro or "transitional" course is not enough -- such a bootstrapping approach and the idea of "growing sysadmin expertise with the system" might backfire discrediting the OS in question more then actual or perceived shortcomings.  At the same time linux certifications suffer from the same "multiple-personalities disorder" that linux itself is suffering from.  Among vendor certifications Red Hat certification looks more objective measure of skills and IQ then Sun's certification.
   
9. After more then fifteen long years, the zeal to build a brave new OS is cooling. The leadership, from Linus Torvalds down to the lowliest kernel driver coder, seems more tired than inspired. The ruling "Linux elite" seem reluctant to make way for younger men. Cynicism due to "make money fast zeal" among the Linux elite during dotcom boom and maladministration of the kernel development further dulled the efforts. Moreover Linux kernel development efforts are spread too thin trying to encompass all the hardware spectrum from laptops to high end servers with just a fraction of resources at hand in comparison with Microsoft. The decision by Linus Torvalds to abandon stable branch of the kernel (previously with great success maintained by Alan Cox) and essentially delegate debugging of the kernel to distributors in version 2.6 does not help too.  That means that outside its major deployment area (low end servers, especially front end web servers) you should expect raw spots. Solaris on X86 is more suitable for midrange servers if and when corresponding applications are available. Solaris is more focused on servers OS, although recently Sun brass also tried to position Solaris 10 to be "all things to all people" and repeat linux mistakes. In case you order Sun X86-based hardware you get an important additional advantage of using a single hardware and software vendor (which eliminates finger pointing), the advantage that is absent for any enterprise linux distribution.
    
10. Contrary to hype, linux does not have advantages over Solaris in the development model. Large scale open source software development projects actually stimulates hierarchical power redistribution with the Great Chairman at the helm and less powerful but no less autocratic "members of Politburo" as the second level of hierarchy, the process of consolidation of power and emergence of elite that long before linux kernel development saga was masterfully depicted in Animal Farm (as reflected in popular quote "all pigs are equal but some pigs are more equal then others").  Like in fluids with certain concentration of salts this process of "crystallization of the elite" is an objective process that occurs independently of the will of the participants and their goals. Moreover reliance on faceless Internet-based communication might amplify some of problems typical for corporate environment and stimulate power struggle at the expense of real work.  Financed by consortium of hardware and commercial software vendors cooperative model used by linux (with the support of enthusiasts from many countries) demonstrated weakness of architectural vision which in turn leads to dominance of imitation at the expense of innovation.  Wrong choice of direction or changes that badly effect stability can propagate all way to the top pretty easily as seeing the whole picture is a difficult task even for the most devoted and talented developers.  With the current level of complexity of the kernel developers including Linus Torvalds looks more like proverbial blind men and an elephant. 
    
    The traditional corporate model with more clear cut lines of responsibility (when a person can be hired for a particular important task or fired for a particular blunder), more concentrated presence of developers at one place, partial suppression of "vanity fair" motives by copywriting the work by faceless corporation and monetary stimulus like stock options might be not as bad as some open source enthusiasts try to depict it.  One of undeniable advantages is that communication between key developers can still be face-to-face.
    
    Due to the age of linux there is an inevitable problem connected with the forthcoming change of the leader, the change that is more problematic and painful that similar change in proprietary unix teams.
     
11. Solaris does not have the danger of proliferation of flavors and as such is vastly superior to both Red Hat and Suse.  This issue cannot be swept under the carpet as there is a real danger to bet on a wrong horse and later face the necessity to support two enterprise flavors of linux in one organization. The leading linux vendor (currently Red Hat) does not occupy very stable position and can be eventually displaced by Novell Suse  or (less likely) Ubuntu which is currently a rising star among linux distributions.  Suse has been tuning kernel for AMD for a few years (they wrote the GCC x86-64 back-end) and now enjoys support of IBM.   All-in-all internal linux fragmentation is the replay of old Unix wars and as such is  a serious threat.   I doubt that enterprise system administrators can benefit from remembering 3 ways of doing things, for example, changing resolution of the screen (one for Suse, one for Red Hat and one for Ubuntu). Just a threat of competing distribution winning at the marketplace over adopted in the particular company (say, Suse vs. Red Hat) somewhat cools enthusiasm for linux.  No amount of hype can hide the fact that the cost of switching from one flavor of enterprise linux to another is comparable with the cost of switching from one proprietary Unix to another: the same level of vendor lock-in and associated problems with re-certification of applications, partial retraining of administrators, etc.  No amount of Linus Torvalds interviews can hide the fact the linux is fragmented into two major enterprise flavors which can be viewed as competing OSes with common kernel.  If you do not understand the value of single version of OS please browse Windows evangelism documents starting from page 9.  While it is highly Microsoft-centric it's pretty instructive as for the role of single standard for the prosperity of ISVs. Note the knockdown of competitors with .NET recently achieved by Microsoft. 
     
12. Linux does has important advantages over Solaris in entry level and mid-range servers deployment areas both due to the availability of cheaper hardware from vendors like Dell (although Sun Opteron-based servers are more or less price competitive and since late 2007 Solaris 10 is officially supported by Dell on its most popular and very good PE1950 and PE2950 servers), wider hardware spectrum supported, as well as due to "home field" advantage (as a platform of development for many open source applications). Those advantages fade as we move to middle range servers. They are completely absent for high end servers were linux is simply not a player (if we discount the role of DOS it is playing in IBM VM/linux mainframes). 
    

13. Historically Linux has far from being impressive compatibility record.  Recently it became better (Suse is the leader in this area), but still abrupt changes are the way of life despite the senior age of the OS (sixteen years are a lot for an OS).  As Steve Ballmer aptly answered the question how one OS beats another availability of source is just one factor: 

    "The way you beat any other competitors: You offer good value, which in this case means good total cost of ownership, right? Because total cost is really, at the end of the day, the issue. And the fact that, quote, Linux is open source, therefore it appears to have a zero price -- that actually made it easier to shine a spotlight on the thing that always mattered anyway, which is total cost. We have a better proposition today, I would think, for total cost of ownership, and we have to offer better value where better value reflects the applications that are available, the quality of the tools. "

    In a way both Linux and Solaris are niche players in the data center stuffed with Microsoft servers and applications and as such should more cooperate then compete.  In X86 space both are definitely riding on coattails of Microsoft as both the cost of X86 hardware and average specifications (including typical amount of RAM) on low and midrange are determined by Microsoft's share of the market.  Large companies now decide about Solaris or Linux, not because they hate one and love another; but because of perceived risks, TCO and how well it will play with their Microsoft part of infrastructure.  That means that a good interoperability is vital and more cooperation between teams is essential.  After all old saying states that the enemy of my enemy is my friend ;-)
    

14. Solaris currently has more technically advanced kernel with much better instrumentation capabilities (due to DTrace and solid OS dump infrastructure) while linux has superior "external" personlaity.  Zones and RBAC are crown jewels of Solaris 10 and nothing in linux currently can match technical superiority of those two technologies. Due to those advantages Solaris 10 is more suitable for deploying complex applications like databases and ERP systems (for example Oracle and SAP/R3).  Dump infrastructure in linux is primitive and buggy. Tracing also leaves much to be desired and far behind the capabilities of DTrace.  Due to better instrumentation with proper tuning Solaris 10 can achieve performance comparative or better then linux on X86 architecture. While linux is definitely fast, rumors about linux being significantly faster then Solaris 10 on Intel/AMD architecture are greatly exaggerated. Actually Solaris has performance edge over linux for applications that heavily use threads.  Availability of  enterprise applications might still be a problem for Solaris 10 for X86.
    
    As for "personality" of OS linux beats Solaris as it looks like more modern Os for administrators and provide them with a lot on non-trivial and important capabilities (better package management, look-back interface, sophisticated LVM, etc).
    
15. Sun has an extremely good and largely deserved reputation in terms of quality of support, training and certification. In those areas it is superior to offerings from Novell or Red Hat although Red Hat has an advantage of keeping training "in-house" while Sun outsourced it and that negatively affects quality.  Novell currently is more democratic vendor as for training and certification  in linux enterprise space (Red Hat has the most expensive training and certification options, expensive even if we are talking about large enterprise financial capabilities). At the same time Red Hat certification is vastly superior to Sun's certification as for reveling real level of qualification and IQ of holders (although I noticed several bad apples here too, it is more difficult to fake by memorizing the material without understanding it).  
    
16. On server level security side Solaris 10 has a substantial lead over any linux distribution and its security mechanisms are less disruptive for applications. Both RBAC implementation and zones are superior to mechanisms used in current linux distributions (with the possible exception in case of Suse AppArmor, which is a very elegant technology indeed).
    
    Solaris RBAC has tremendous value as a security mechanism perfectly suited for the large enterprise environment. In combination with zones RBAC represents a unique method of preventing "root sharing hell" by ensuring real separation of duties. Solaris zones essentially allows application owner to control it own lightweight virtual machine and as such greatly reduce conflicts in access control in Unix environment.
    
    Recently Solaris RBAC also became one of few ways to channel large part of SOX compliance efforts in a constructive way and limit the negative influence of "SOX socialism" on large enterprise IT environment. For RHEL 4 no amount of hardening can match the security of Solaris with applications running in zones and well structured RBAC "separation of duties". Suse AppAmor does has promise, while RHEL security infrastructure suffers from overcomplexity and due to overcomplexity actually is a weak point of distribution (cases of systematic switching it off in production servers are not rumors).  This fact, combined with the necessity (and dangers) of more frequent patching for linux means that maintaining the same level security of servers on linux servers will always be more expensive for large enterprises then maintaining the same level of security on Solaris 10 servers be it X86 servers of UltraSparc servers.  Of course here like in most other areas the qualification of staff  is more important factor then differences between those two OSes. 
    

As such it is more hostile to linux then Solaris and from system administration standpoint its "linux affinity" is more a myth (and a very shrewd PR move) then a reality.  AIX 6 can probably be called AIX5S as instead of fake linux affinity it demonstrates real Solaris affinity with zones and RBAC: imitation is the most sincere form of flattery.

It even makes one step further in development of Solaris zones concept as it might implement "live" zone migration before Solaris (it logically requires 10G links). Hopefully AIX6 will narrow the gap in system administration with linux and Solaris.  It would be nice if IBM diminished "non-essential" but annoying system administration idiosyncrasies including, but not limited to, fixing incompatible error reporting subsystem (syslogd is far from being perfect but can be extended to fix the most glaring shortcomings like was done in syslog-ng), adding utilities like pgrep and pkill for /proc pseudo file system, as well as normalized command line utilities for creation and modification of user accounts. 

I would like to stress that Solaris 10 really can claim "linux affinity". My feeling is that mixing Solaris and linux in enterprise environment produces the least amount of mental stress and overload for system administrators in large enterprise environment in comparison with any other combination of Unix flavors.  In this respect the level of independence of "linux press" with its unconditional support and praise of AIX raises some questions.  AIX is really interesting and in my opinion underappreciated Unix flavor (its virtualization capabilities are a real masterpiece of software engineering in best "old IBM" traditions of technological innovation ) but claims of "linux affinity" are open to review and look somewhat suspicious for any unbiased observer.  It is the most idiosyncratic flavor of Unix in large enterprise space.

Both linux and Solaris, each in its own way, are continuing BSD tradition of "Unix as a culture" and both are key parts of Unix renaissance. This paper suggest that Solaris as a brand did quite a lot of good both to Unix as technical artifact as well as  Unix as a cultural movement (including linux), so attempt of zealots to blackmail Solaris are misguided.  Early years of Solaris are closely associated with Bill Joy, one of the most influential and creative figures in the history of UNIX, and it is undeniable that Solaris introduced many important innovations in Unix, the record that linux simply cannot match.

Additional advantage of Solaris 10 is that in large enterprise environment it can serve as a "fall-back" solution for botched linux deployments in case those deployments are done on Sun X86 hardware (of Solaris-compatible hardware -- many HP AMD-based servers are certified for Solaris 10).  The quality of Sun's X86 servers is high and the prices are competitive.  The same is true for HP  Solaris-compatible servers.  There is no reason why you should put all eggs into one basket and using Sun X86 servers provides for large enterprise IT architects some needed flexibility and thus increase that chances of success of the complex projects.  While many of open source enthusiasts would prefer linux for everything they do, nobody can deny that "stability is a sign of professionalism" and I would add that the compatibility is an important sign of professionalism too.

With Solaris 10,  like in the cases of Red Hat and Suse, the cost of support is extra, but, in comparison with RedHat,  Solaris 10 is more reasonably priced. Security patches are free which makes Solaris similar to Windows (the latter paradoxically is very competitive with linux if we are limiting ourselves to enterprise distributions with paid annual support requirements such as Red Hat and Suse).

In a way Sun make a very shrewd move by opening Solaris. It convincingly demonstrated that it can compete in free and "maintenance fee-only"  Intel-based OS space and, moreover, that in certain areas it can beat other competitors in this space. As marketplace is not always rational we will soon see it this is enough or not. anyway, from technical standpoint currently Solaris 10 is the leader among Unix flavors in large enterprise space and while you don't have to love the leader (or respect technical quality; this is an unfashionable thing in modern IT ;-), some experimentation and acquiring the working knowledge of the system would not hurt even for the most avid linux supporters. They might be surprised what they find...

If we compare large technical movements to religions it is clear that those who love Unix should not absolutize the differences between flavors and be tolerant toward particular expression of the same set of basic values (Unix culture) in a slightly different form.  Historically immature behavior and claims of superiority is typical for younger, less established sects  ;-).

Generally programming is not  a place for blind devotion. As John Stuard Mills aptly put it many years before computers became available:

Very few facts are able to tell their own story, without comments to bring out their meaning. The whole strength and value, then, of human judgment, depending on the one property, that it can be set right when it is wrong, reliance can be placed on it only when the means of setting it right are kept constantly at hand. In the case of any person whose judgment is really deserving of confidence, how has it become so? Because he has kept his mind open to criticism of his opinions and conduct. Because it has been his practice to listen to all that could be said against him; to profit by as much of it as was just, and expound to himself, and upon occasion to others, the fallacy of what was fallacious. Because he has felt, that the only way in which a human being can make some approach to knowing the whole of a subject, is by hearing what can be said about it by persons of every variety of opinion, and studying all modes in which it can be looked at by every character of mind. No wise man ever acquired his wisdom in any mode but this; nor is it in the nature of human intellect to become wise in any other manner.

The steady habit of correcting and completing his own opinion by collating it with those of others, so far from causing doubt and hesitation in carrying it into practice, is the only stable foundation for a just reliance on it: for, being cognizant of all that can, at least obviously, be said against him, and having taken up his position against all gainsayers knowing that he has sought for objections and difficulties, instead of avoiding them, and has shut out no light which can be thrown upon the subject from any quarter -- he has a right to think his judgment better than that of any person, or any multitude, who have not gone through a similar process.

As somebody aptly joked about Linux on one of Information Week forums: "Have you people tried writing poetry about your lame hatred towards Microsoft? " That's joke is fully applicable to the subset of Solaris haters among linux aficionados.

Created Jan 2, 2005.  Last modified: June 05, 2008