Softpanorama (slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

The cost of support

Cost of support in a long run is as important or more important as upfront hardware costs and can exceed them making it the largest component of TCO.  Yes, linux has several free distributions (Fedora, Debian, Ubuntu, etc) and offers better hardware support for workstations and low end servers. Due to this it is perfectly suitable for fighting corporate red-tape (most recent linux distributions usually runs on a typical corporate desktop or Intel 1U server "out-of-the box"; that makes it ideal for quick prototyping that can help to avoid costly deployment mistakes.)  But TCO of linux in large organization is a factor that needs to be carefully evaluated. In this sense linux is far from being free. 

Enterprise-class distributions (Red Hat and Suse) are not cheap and five years TCO of, say, 100 Red Hat servers is nothing to boast about. Actually for pure infrastructure servers Microsoft might have slight advantage over "enterprise open source OSes" as it charges a fixed license fee (patches are free and support contacts are optional and can be bought on "per incident" basis). We assume that infrastructure servers do not require more then 10-20 user licenses (mainly for administrators and operators).   IMHO less than third of a five years OS TCO is in upfront costs (e.g., initial price of software license plus initial support contract), while the rest is in year-to-year support, training and, especially,  "know-how" costs. Cost of support calculations generally should include the following:

• Re-training costs. In case of adopting linux they can be substantial for a large geographically distributed company with data centers in many different countries. Even if we assume that it is possible to save $2K per server, or $200K per 100 servers (hundred is a typical measurement unit in large enterprises) those hardware savings are always offset by the cost of additional training and certification (let's say $2,500 per course with 20 additional courses required per each hundred of new installed servers for five year period). In case there is a necessity to hire even one additional administrator, all the hardware savings simply evaporate after a couple of years.
   

• Additional costs due to the non-uniformity of the server park. There are some additional software maintenance costs due to multiple (and lower volume) contacts with each hardware and OS vendor as well as the additional costs associated with a "new style" Unix support cost structure typical for linux (for example, Red Hat has rather expensive support contracts for RHEL AS line ).
   

• Personnel related costs  including costs of (re)education and availability (and uniformity across the industry) of certification as well as "know-how" costs.  In the areas of training and certification linux has a distinct disadvantage as it is split into two competing enterprise distributions, each with its own training system and certification program. At the same time Red Hat certification looks superior to Sun's certification and looks like more objective measure of a person knowledge and IQ, less easily undermined by cheater who use memorizing of the material and set of typical questions as a substitute for real knowledge.
  
  As we already mentioned above, the ability to avoid "excessive Unix diversification"  largely depends on the quality of high-level IT management that should act as a counterbalance to the internal political forces favoring particular distribution (for example large enterprises with significant European presence usually have forces favoring Suse over Red Hat). With a typical quality of high-level IT management it is not unreasonable to assume that many large enterprises might end with deploying both major enterprise flavors of linux (Red Hat and Suse) driving the education and certification costs higher.
   

• Solaris has Dtrace that can lower the cost of support and troubleshooting. It is a very powerful tool that makes Solaris a class of its own but requires qualified personnel to use, though.  Note that this ability to tune is different from the ability to modify the source code as one of the advantages of open source but it really does not matter much in large enterprise IT ecosystem. Usually only early adopters are in the game of tweaking the source code. If product reaches maturity (and codebase get the level of complexity typical for mature enterprise-level products) the percentage of users who dare to tweak the code (and thus totally own the support of the codebase) quickly diminishes to the level approximating zero.

  DNS can serve as a good example of this trend. BIND, the open source DNS server that is maintained by ISC, has the highest market penetration (approximately 80% I think) of all DNS servers on the market, one of the highest among "enterprise ready" open source products. But the amount of people who tweak bind code is probably less that a one tens of one percent of userbase. Most changes are related to tweaking in order to improve the performance of critical DNS servers, if any.  If something goes wrong, Dtrace serves as a comprehensive troubleshooting tool that greatly helps to understand real situation and avoid false moves. That means that availability of Dtrace makes Solaris preferred platform for running bind in comparison with Linux.

DNS deployment also demonstrates another an interesting problem for open source in large organizations. Sometimes the cost of support of open source application can be so low that it hurts the organization. Free downloads and deployment on low cost Intel boxes do not require a conscious investment decision and as such often are undetected by any IT decision makers. As a result the associated implementations may promote wrong product, be underfunded or completely "grass-root" based. Even if the product is right for the organization the implementation can contain severe configuration errors, have architecture that leaves room for improvement and/or even threaten the stability of mission-critical systems due to its position in the infrastructure. For example, DNS, while can affect the stability of any other application, sometimes completely disappears from the management radar screen and enter the twilight zone when it just exists by-and-large due to the benevolence of staff.  Not that large organization IT infrastructure are foreign to useless (and expensive) products, but this threat of stealth deployment of useless product or incorrect deployment of useful product  needs to be accounted for.

That might be one reason why some DNS servers in large organizations are quite often misconfigured and insecure. If this is true, then deploying DNS on linux quite counter-intuitively might be a bad decision considering the importance of well-functioning DNS for any large organization. Solaris is more closely associated with paying for applications support and formal commercial application support infrastructure. Generally it can be easier to get proper funding for projects deployed on Solaris (or AIX or HP-UX). Just because of this purely psychological moment it might be a better platform for deployment for open source Internet infrastructure-related applications, applications that otherwise can remained under funded and as a consequence can be misconfigured and/or under supported.

Paradoxically the example of DNS also shows the risks when an infrastructure-related open source product is adopted and embraced on the grassroots level of an organization but despite its importance is off the radar screen of IT decision makers because the acquisition cost is zero and the support cost is formally non-existent (voluntarily absorbed by staff due to their enthusiasm about particular open source product).  As this situation is not sustainable, there are certain associated risks in grassroots adoptions of open source in large organizations if they are done without reserving appropriate support costs in the budget.  

Created Jan 2, 2005.  Last modified: June 05, 2008