Softpanorama
(slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

[an error occurred while processing this directive]

Solaris Competitiveness on Low End Servers

If we are talking about low-end servers here generally linux has an upper hand due to ability to run on wider range of hardware as well as relentless improvement of Intel-compatible CPUs and chipsets. Essentially hardware-wise linux rides Windows bandwagon. As an OS Linux more quickly improves on low end as low-end servers are its core specialization and is one area where it achieved significant penetration (all four major low end servers vendor ships Linux preinstalled on their servers). As version 2.6 of Linux kernel matures it might be able to increase its share among low end servers, especially in key linux applications areas (web-servers, departmental file servers, DNS and mail servers, etc) despite strong competition from Solaris, Windows and BSD (Apple should now be considered as a vendor of an OS belonging to BSD-line of OSes and also offers low-end servers).

Please note that the validity of price-related arguments below is limited to mainland USA. For other regions prices can be very different. For example some software companies (Microsoft is one example) often charge higher prices in Eastern Europe in comparison with mainland USA where prices are very reasonable (price discrimination under the disguise of compensating for piracy ).

It's important to understand that on low-end servers hardware prices and hardware compatibility list greatly influence competitiveness of the OS: hardware prices and the hardware compatibility list and reliability are as important as OS prices, support prices and reliability. So our discussion will be fuzzy here: we will jump from OS to hardware and hardware compatibility issues multiple times without any warnings. For example Sun's questionable policy of supplying servers mostly with 80G drives (probably in a hope that customers will buy NAS for anything larger then 80G) is a serious downside for Solaris on low end as it limits that capabilities to use Solaris with Sun's hardware -- by definition the most compatible and thourously tested brand of hardware for the OS. Of course you can switch to HP. Several Proliant server are Solaris-compatible (see HP.com - Solaris support on HP ProLiant Servers )

Generally on low end you should not expect good customer service. That's an illusion. Margins are thin. Many giant corporations moved its low end customer service to bored, ill-treated, underpaid people desperate to move on to better jobs. And you would definitely prefer to use as much Web applications for creating the final configuration and quote as possible. Here Sun have some problems. Sun does not want to deal with you directly unless you are buying $20K or servers or so. They will try to push you to local VARs and integrators like CDW (actually Sun here is not the worst; situation with HP is even worse). You still can buy certain "fixed" configurations from the WEB site using corporate credit card (few organization approve usage of corporate credit card for such purpose). Generally Sun's ecommerce WEB site is so primitive that I wonder how people are buying low end servers from them. It has an indelible mark of unprofessionalism.

Generally Sun's ecommerce WEB site is so primitive that I wonder how people are buying low-end servers from them. It has an indelible mark of unprofessionalism

Dell predominantly does direct sales and among four major low end server vendors (Dell, IBM, HP, and Sun) they are the only one with usable Web site where you can price your order without speaking to customer rep. Also in case of Dell any large company usually already have a rep to talk to and he is ready to discuss detail of the quotes that you already got on the WEB using their small business division and get large company discounts that you particular company is entitled due to volume agreements with Dell. They are good partners even if you are buying is one server. The fact that Solaris 10 is not supported by Dell undermines Solaris 10 rating on low end.

Generally comparing OSes on low end servers the following factors look as the most important:

Home field advantage
General conservatism of the large enterprise environment
Licensing and Support Costs
Comparative low end hardware costs and capabilities
- Remote Management
- Custom Automated Installation
Hardware compatibility
- Peripherals and Disks Selection
Efficiency
Comparative security and virtualization capabilities

Returning to OS comparison, linux is attractive on low end, but not due to its low cost (for large enterprise this is an urban myth, as we will show below) and unsubstantiated claims like better security or, god forbid, better kernel quality. The key is application support (especially open source applications support) and drivers availability. In essence linux is Microsoft Windows of Unix: popular OS of average quality running on the most popular hardware and the most price effective platform in existence and as such it is and always will be the most insecure flavor of Unix in existence. Like in workstations Linux owes much of its success on low end to Microsoft as it is Microsoft that is owner of PC workstations and Intel servers standards, the standard that it releases for free to all major manufactures. Sun with its UltraSparc line (much like IBM with Power5 line) is more like an icebreaker that need to charter its own course in cold waters.

Home field advantage

For many open source applications (as well as some commercial applications) as it is not the OS itself but the applications that determine the attraction of linux or Solaris for large enterprise customers. Here linux (actually only Red Hat) is definitely preferable as the number of supported precompiled open source applications in Solaris while reasonable and includes all major application is still a tiny fraction of packages supported for Red Hat.

General conservatism of large corporate environment

In many cases RISK-based servers (not necessary Sun) are so entrenched that to replace them with Linux of Intel (or Solaris on Opteron) is very costly despite huge price advantages of X86 architecture. I know many cases where enterprise bought comply outdated and very expensive HP boxes (one example of such replacement is the replacement of multiple boxes with new HP servers that have 550MHz PA-RISK CPU, two 73G drives and 6G of memory. Each costs over $6K each; note that HP charges for CD-ROM for such a box $500 -- the same price that a low end server from Dell would cost). Often the environment is so conservative that migration to more modern systems can be done only after big corporate earthquakes like mergers and acquisitions. Also realistically not all servers need 3.2GHz CPUs or couple of them. Many can work well with much less horsepower if they have adequate amount of memory installed.

Support costs and electricity costs. Both support costs and electricity costs significantly influence the total cost of ownership of low end servers. For example for a typical lifespan of low-end server (five years) the cost of support $500 per year is equal to halve of the cost of the $5K server (That explains attractiveness of CentOS as an alternative to Red Hat for small business.) Electricity consumption makes another half.

I would like to stress it again that for a low-end server the flexibility of configuration (for example the ability to install wide range of hard drives on the server) is of paramount importance and Sun currently is bad (or average depending on your point of view) in this area (see below). Low quality of Sun's eCommerce site partially disqualifies Sun from many low-end server deals despite high quality of its hardware (especially X4100 servers as well as T1000 and T2000 servers). I would like to stress that in my view Sun's Achilles' spot on low end is harddrive selection, as we will see below.

Licensing and support costs

The licensing terms for Red Hat looks really menacing for low end servers. Here is the relevant quote from the licensing agreement:

4. REPORTING AND AUDIT. If Customer wishes to increase the number of Installed System, then Customer will purchase from Red Hat additional Services for each additional Installed System. During the term of this Agreement and for one (1) year thereafter, Customer expressly grants to Red Hat the right to audit Customer's facilities and records from time to time in order to verify Customer's compliance with the terms and conditions of this Agreement. Any such audit shall only take place during Customer's normal business hours and upon no less than ten (10) days prior written notice from Red Hat. Red Hat shall conduct no more than one such audit in any twelve-month period except for the express purpose of assuring compliance by Customer where non-compliance has been established in a prior audit. Red Hat shall give Customer written notice of any non-compliance, and if a payment deficiency exists, then Customer shall have fifteen (15) days from the date of such notice to make payment to Red Hat for any payment deficiency. The amount of the payment deficiency will be determined by multiplying the number of underreported Installed Systems or Services by the annual fee for such item. If Customer is found to have underreported the number of Installed Systems or amount of Services by more than five percent (5%), Customer shall, in addition to the annual fee for such item, pay liquidated damages equal to twenty percent (20%) of the underreported fees for loss of income and administration costs suffered by Red Hat as a result.

If your organization wants to run any of the RHEL-certified enterprise applications and receive support from the software vendors you have no choice but to buy RHEL from Red Hat on the terms it specifies. In this sense Red Hat is a regular proprietary OS with rather strict licensing and all PR ink about "evil proprietary Microsoft" and "open source leader" (with Windows being cheaper then Red Hat if you factor in the support costs :-) should probably be spend for more noble purposes. Red Hat business is based on the fact that GPL permits arbitrary restrictions on binary distribution rights as long as source code is available. In this sense Red Hat is a regular OS vendor that provides access to its binaries for a price. The only unique feature is that due to GPL it is suffering from leaks due to availability of source (CentOS).

You can read more about my views on GPL licensing and potential problems with it in ebook Labyrinth of Software Freedom (BSD vs GPL and social aspects of free licensing debate).

Taking into account that for each instance of the server you need a valid license and annual support costs are integral part of the cost of ownership. Here are approximate annual costs [RedHat2006]:

	Red Hat Enterprise Linux AS		Red Hat Enterprise Linux ES
	Premium Edition	Standard Edition	Standard Edition	Basic Edition
Price Per System	$2499	$1499	$799	$349

As we can see there is no cost advantage for Red Hat in comparison with Solaris 10. Most vendors sell the system with more expensive ES standard edition, not with Basic edition but this probably can be negotiated. Anyway the price of support per system is approximately the same. The main difference is that you can use Solaris without support contact and security patches will still be free, but you cannot do the same for Red Hat.

Here is how recent (January 19, 2007) article by Timothy Morgan compares support prices for two OSes [Morgan2007]:

Under the new pricing scheme, Sun is charging based on the number of sockets in a system and the type of processors--X64 or Sparc--used in the system. Like Novell, Sun is giving a discount if customers ink a three-year contract--something that Red Hat doesn't do on its online price list, but which it certainly does in practice. Sun is offering a $49 annual developer support contract for nerds who just want email support and who don't want to pay much for support, since they know plenty themselves. This is not really a data center product. The entry support offering for Solaris 10 on servers now is the Basic plan, which costs $240 a year on machines with up to two sockets, whether they are Sparc or X64 sockets. The Basic plan includes 30 days of telephone support with real people and a year of Web-based support. A three-year contract costs $648, which is a 10 percent discount over the cost of buying three annual licenses in a row.

... ... ...

Red Hat's pricing is simpler, and is, like Sun's, based on the scalability of the server. Red Hat Enterprise Linux 4 comes in two flavors for servers--the ES variant for machines with up to two sockets and the AS variant for machines that have three or more sockets. RHEL 4 ES Basic support, which comes with 30 days of installation support and one year of Web-based support and access to the Red Hat Network for patch updates, costs $349. RHEL 4 ES also comes with Standard support, which covers the 12 hour, five day business hours and provides four-hour response times on problems; it costs $799. On RHEL 4 AS, which is more scalable, this same Standard support costs $1,499 and there is no Basic support option. And finally, RHEL 4 AS Premium support, which costs $2,499, provides one-hour response on tech problems and all the other goodies on a 24 hour, seven day basis. There is no Premium support option for RHEL 4 ES.

As we can see on entry level Sun is providing 30 days of telephone support which Red Hat does not match. And that's a very important difference.

Note: if you buy hardware from Dell you will get 50% discount for RHEL standard edition and that makes RHEL quote competitive with Solaris at least for the initial three years of service covered by the subscription. Both Red Hat and Solaris charge for each additional CPU (so multicore chips benefit from this treatment). Solaris scale is flat. Red Hat doubles the support costs after two CPUs (you need AS version). That makes service plans for Red Hat competitive with Sun only on low end (up to two CPUs).

Comparative low end hardware costs

First let's discuss hardware costs as on low-end they influence OS selection independently of OS qualities. You never see in linux advertisement pages in major trade magazines the fact that a typical cost of Red Hat Enterprise Server ES license ($800) is big enough to make it less attractive then Windows. It is difficult to complicate this cost with the lower cost of low-end server hardware (Dell is an exception here due to 50% discount that you get on Red Hat ES distribution). That makes switching from existing flavors of Unix less attractive as there are no significant savings here. Also it makes attractive using other flavors of Linux (that's why HP recently started to support Debian) contributing to fragmentation of Linux market and creating modern variant off Unix-hell.

That means that for large enterprises low end servers with linux is almost as expensive as Solaris on UltraSparc and Opteron if the cost of support contract is factored in (let's say for five years -- typical lifespan of low-end server). And even with Dell prices if you factor in 5-year cost of support subscription 349*5=$1745, Red Hat is approximately twice more expensive Windows Server 2003($800) with default user licensing (5 CAL). Still if you buy low end servers from Dell the truth is that they are price and performance comparative with best Sun offering despite the additional cost of RHEL.

Differences in hardware prices for Sun T1-based and Opteron boxes and low end Dell boxes (Dell beats cost-wise both HP and IBM so they can be safely ignored here) currently are less pronounced then two years ago and this difference only slightly exceed the cost of Red Hat 4.1 ES license ($800). Here is a brief comparison using prices as of June 27, 2006:

	Server	Memory	Disks	Price
1	Sun Opteron 1U server X4100 with two model 275 CPUs, redundant power supply, 4 Ethernet ports, Solaris Preinstalled, one year support contact, build-in iLOM	4G of memory,	2*73G SAS 10K RPM	$4,400
2	1U Sun Sparc server (T1000) with 1 T1 CPU (1.0 GHz, 6 Cores), Solaris preinstalled, one year support contract, build-in ALOM.	8G of memory	1*80G 7200 RPM SATA	$5,700
3	Dell PowerEdge 1950 1U server with one Dual Core Intel Xeon 5080, 2x2MB Cache, 3.2GHz, 1066MHz FSB, Red Hat 4.1 ES one year support contract, redundant power supply, 4 Ethernet ports (two on the motherboard plus additional two ports card), and Dell Remote Access Controller 4	4GB 533MHz (4x1GB Dual Ranked DIMMs),	2*73G SAS 10K RPM	$4,700

If you do not have Dell discount the cost of Red Hat license makes the advantages of switching from UltraSparc on hardware side and from Solaris OS side less compelling. This opinion was actually voiced by Dell executives as early as 2004 for a different market segment and they should be commended for getting this discount from Red Hat for thier hardware customers [News.com]:

Red Hat needs to lower its prices, or risk losing customers to free versions of the open-source operating system, the Dell executive who oversees the partnership with the Linux seller said Tuesday.

"We believe Red Hat Enterprise Linux 3, for the small and medium-sized business market, was out of the price range of these customers," said Judy Chavis, director of business development for Dell's enterprise product group.

And Dell has the marketing muscle to make its opinions clear. Indeed, Red Hat's pricing was instrumental in Dell's decision to sign its October pact to sell Novell's SuSE Linux. "It was definitely a factor in us working with Novell," Chavis said in an interview here at Oracle's OpenWorld conference. "Novell was able to step in and offer us that price point."

Red Hat declined to comment for this report except to say that Dell is a strategic partner.

Of course you can always use CentOS instead of Reh Hat but that's not what large enterprises typically do.

Hardware and OS capabilities on low end

Now let's talk about OS capabilities. Both on UltraSparc and Opteron platforms combination of Solaris and Sun's hardware provides some unique capabilities like self-healing, better security then linux [Bezroukov2005b] and equal or better stability. Therefore while for the same price you can get more capable server from Dell then from Sun, some of advantages of Dell hardware can be compensated by advantages of Solaris over Linux, especially in UltraSparc line of servers based on T1 CPU.

The ninth generation of Dell's PowerEdge server that was launched in July 2006 are very capable low end servers with good selection of hard drives and peripherals. They represents direct competition to Sun's 4100 and 4200 line of servers. Intel's improved "Woodcrest" Xeon Core processors (Xeon 51xx) which offer better performance per watt than the Dempsey chips. The new PowerEdge 1950 uses the 3.2 GHz processor with a 1.07 GHz front side bus on high end and 3 GHz CPU uses 667 MHz front side bus on low end. Each Dempsey core has 2 MB of L2 cache, and interfaces with the Intel 5000X chipset cable of supporting fully buffered DIMM memory (533 MHz or 667 MHz), and has eight memory slots. The densest DIMMs is 4 GB so the server is capable handling up to 32 G of memory across four processor cores (with a potential for eight software threads if customers turn on HyperThreading). The machine can support two 3.5-inch hard drives or four 2.5-inch hard drives, and has room for a two PCI-X or PCI-Express slots. It comes with dual Gigabit Ethernet ports, with room for optional ports.

As for benchmarks, older Jan 2006 Anandtech's comparison gives you the idea of performance you should expect Older Xeons (before 51xx line have low benchmarks then Operons so you can expect ~20% better reading for Opteron based servers. Please note that in large enterprise environment low end servers the most often deployed in web server farms:

Specweb2005

"SPECweb2005 emulates users sending browser requests over broadband Internet connections to a web server," De Gelas explains. "It provides three new workloads: a banking site (HTTPS), an e-commerce site (HTTP/HTTPS mix), and a support site (HTTP). Dynamic content is implemented in PHP and JSP."

Sun Fire T2000: Processors: 1x 1.2GHz UltraSparc T1; Power Dissipation CPUs: 72-79 W; # of Cores: 8; # of Active Threads: 32 Score: 14,001; % Score: 289%

IBM p5 550: Processors: 2x 1.9GHz POWER5+; Power Dissipation CPUs: 320-360 W; # of Cores: 4; # of Active Threads: 8; Score: 7,881; % Score: 162%

IBM xSeries 346: Processors: 2x 3.8GHz Xeon; Power Dissipation CPUs: 220-260 W; # of Cores: 4; # of Active Threads: 4; Score: 4,348; % Score: 90%

Dell 2850: Processors: 2x 2.8GHz DC Xeon; Power Dissipation CPUs: 260-300 W; # of Cores: 4; # of Active Threads: 8 Score: 4,850 % Score: 100%

We will discuss energy costs later but they should be noted as the advantage of T1 line of servers in large corporate environment. Another important capability that is unique to T1 line is that similar to high-end Sun SMP systems, the UltraSPARC T1 can be partitioned under Solaris 10. Thus, several cores can be partitioned for running a single or group of processes and/or threads, whilst the other cores deal with the rest of the processes on the system.

That's why despite the fact that the quality of both Solaris 10 on Opteron and Sun new Opteron servers line (X4xxx) is very impressive (they hold several industry benchmark records), I think that the most competitive segment of current Sun offerings for large enterprises on low end are new T1-based UltraSparc offerings. The extra value of the Open Boot PROM and extra level of security (the latter is not limited to UltraSparc but to the whole class on non-X86 CPUs, for example Power5) might be well worth the extra cost of the T1000 or T2000. Proven reliability and manageability of tried and true UltraSparc offerings should also be factored in.

Like its UltraSparc counterpart, 64-bit version of Solaris kernel for Opteron is able to utilize the advantages of new EMT64T architecture with a MMU that has a "no execute" bit on a memory segment and thus has a huge security advantage over competitors. No matter whether we generally prefer one OS or another it is important to understand that no level of linux hardening can compensate for the absence of protection from stack overflows, the protection which is a standard feature of Solaris 10 on Opteron. This is an advantage of Solaris on Opteron, the advantage that it shares with Solaris on UltraSparc and that no enterprise customer can ignore.

As for design quality both X2xxx and X4xxx are among the top offerings on the market. In 2006 Sun Fire X4100 servers (together with Sun's Ultra Series Workstations and Sun Fire T2000 server) received a bronze Industrial Design Excellence Award (IDEA) in the Design Strategy category. This prestigious award is co-sponsored by Business Week magazine and the Industrial Designers Society of America (IDSA) for the best and most innovative product designs of the year.

Still for organizations with large installed UltraSparc base the pain of supporting slightly different platforms and administering slightly different OS (even if it is called Solaris :-) should be taken into consideration against potential lower cost and higher productivity of Opteron-based servers. That is the just another aspect of adding flavors of Unix to the enterprise OS mix, the factor that makes the introduction of linux into large corporate environment such a mixed blessing.

Remote Management

Managing servers from a remote location is often mandatory in today's large enterprise environment. IT administrators must be able to manage servers in secure data centers or in locations that have no or minimal administrative IT staff. Such scenarios require remotely performing all server management operations and responding to server-down situations.

X4100 servers have built it network management port with secure remote access using Web Interface over SSL or CLI over SSH. They support full lights-out remote management using remote keyboard and mouse, remote CD/DVD-ROM and remote graphics. Sun's integrated service processor (iLOM) is very capable. You also get 4 gigabit Ethernet ports plus a dedicated 10/100 for the iLOM, plus you have two PCI-X Low Profile slots, redundant power supplies, redundant hot-swap fans, etc. To save energy default configuration of T1000 does not have CDROM drive or video card so you need to use Jumpstart. Also Sun N1 System Manager is free and increases the value of the system: it really simplifies hardware monitoring and software deployment.

All UltraSparc servers has Sun Advanced Lights Out Manager (ALOM). The ALOM System Controller hardware is embedded in the server and functions completely independently as a separate embedded computer. The ALOM can monitor hardware status, perform hardware resets, power on and off, and can send e-mail if alarms are triggered. You can switch between the ALOM Command Shell and the OpenBoot PROM Prompt. To switch to ALOM, type #. (pound-period) on a serial console. After that you can configure your management network interface and many other parameters. The network interface variables specify the network settings that ALOM uses across its Ethernet connection at the network management (NET MGT) port on the server. ALOM functions allow for monitoring, logging, alerting and for basic control of the system. ALOM is particularly useful for remotely managing a server in a typical "lights out" environment. Version 1.5.5 provides out-of-box network management interface that includes [Sun2006b]:

Serial interface (CLI)
10 Mb Ethernet interface (telnet CLI)
Host side interface/admin utility
Host side monitoring (disks, fans, power, temp, PSU, FRUID, LED, hotplug devices)
Host control (power, reset)
Server watchdog XIR: (OS monitor/restart/XIR)
Self watchdog/restart
Enhanced events (ALOM events, host events, hotplug devices)
Logging events to ALOM (including nvram and syslog)
Logging host console stream output
User configurable delivery of events to ALOM CLI, email, and syslog
Optimized boot time
Hardware environmental monitoring/management

Please note that virtual CDROM capability is absent and that's an important deficiency. You can use Jumpstart instead. Also, admins can run Sun Management Center (MC) on a system that has a graphics card and administer the server remotely that way. The MC provides a centralized GUI for most administrative functions through a Web interface, and integrates with the ALOM for hardware control when the proper agents are installed.

Dell has superior capabilities with Dell Remote Access Controller 5 (DRAC 5) which is an optional PCI card ($299) which comes with its own processor, graphics chipset and network port. With full administrative access, you can shut down the OS, recycle power and turn off or switch the server back on. It also support virtual floppy and CD-ROM. That means that the system can be booted from an image stored on another system.

DRAC 5 is more capable then Sun ALOM: along with CLI interface DRAC firmware has an embedded Web server so that you can connect to it from a any PC that has a Web browser. Web browser interface works very similar to VNC but currently in working with linux it has problems with mouse: it is quirky and not very pleasant to use. Also unlike working with Windows you see two mouse pointers: one for "external world" and the other for "internal world" (if you connect to Windows server those two mouses are merge into one all the time). That creates quite a lot of confusion as as moving "wrong cursor" and clicking on it can result in unpredictable behavior if "right cursor" accidentally also was position of some kind of button or menu.

But to be fair you need to understand that this is a recovery console, not your every daytool (althouth it can be used as such); so standards are more relaxed here and the fact that it works with GUI is a significant advantage. It, for example, greatly simplifies reconfiguration of network after server move as you can use RHEL GUI widgets for that (much like in Windows) and do not need to remember all the intricacies of linux command line commands like ifconfig that are slightly different from classic Solaris version (FSF behaves a lot like Microsoft in the area preaching a variant of "divide and counker" theme for software world -- embrace and extend; see GNU echo parody).

Dell also have integrated deployment and change management solution called Dell OpenManage 5.0. With DRAC5 it delivers comprehensive hardware deployment, monitoring and management with features including [Itreseller2006]:

Integrated hardware and software management with partner applications like Altiris Server Management Suite, Microsoft SMS and MOM, and Novell ZENworks Linux Management - Dell Edition, providing single console management of enterprise resources
Dell OpenManage Connections, which allows comprehensive hardware monitoring within industry management frameworks
IMPI 2.0 and SMASH, which deliver simplified management capabilities today and in the future via a standardized management infrastructure
Improved Linux Integration with full support for Novell SuSE Linux SLES9
Improved integration between Dell OpenManage and VMware ESX 3.0, allowing simplified monitoring for customers' hardware devices dedicated to virtual machines.

Custom Automated Installation

There are still advanced administrative technologies that are more commonly used in large corporate IT then in startups (outside ISPs). One of them is custom automated installation. This area is represented by Jumpstart in Solaris and its Red Hat clone Kickstart. The latter was first released with RedHat Linux version 5 in 1998 (not documented) and later was also implemented in Suse. Both Jumpstart and Kickstart permit automation of key elements of installation process including but not limited to:

Language selection
Network configuration and distribution source selection
Keyboard selection
Disk partitioning and filesystem creation
X Window system server configuration
Timezone selection
Selection of an (initial) root password
Selection of packages to install and to remove
Initial hardening of the system.

Solaris Jumpstart has also a variant called webstart in which you can create an image of the system and them replicate it on as many systems as you wish, introducing customarization on the level of already fully configured system. The image of the system is called flash archive and it can be installed. I do not know about similar capabilities of Kickstart.

Flash archive are a very important productivity tool and can save 80% of the time in replication of complex server setups to remote locations.

Hardware Compatibility

Support of the OS by all four major low end server vendors including Sun is of primary important and linux beats Solaris in this respect. Hardware compatibility-wise, while linux cannot compete with Microsoft in the range of hardware it supports it is king of hill in this area in Unix world. While Sun is the most competitive among RISK vendors (and generally on low end RISK servers leaves IBM and especially HP in the dust), Sun's Achilles spot for low-end servers is harddrive selection. Still this deficiency, although acute for some customers, might not be that a big deal: currently in large enterprise environment linux is used only for rather narrow set of roles with web server farms, DNS and mail as a primary one. For such roles the variety of hardware does not pay so much a role and virtualization capabilities are very important. That means that Solaris can and should compete well with Linux both on UltraSparc and Opteron despite more limited hardware selection option. As for virtualization, now this technology also become fashionable and fashion as well as "political correctness" are very important forces in large enterprise IT (note, for example, how many useless IDS sensors a typical large enterprise deploy).

Peripherals and Disks Selection

In the past the main problem with UltraSparc low-end servers on SPARC was very limited list of disks and additional hardware they support (hardware compatibility list). Also the prices for most of those component are really high (DVD reader for V210/V240 has list price of $400). I remember buying 146G instead of 300G SCSI disks for V210 just because 300G Seagate drives were not in hardware compatibility list (they might well work OK, after all they are very similar SCSI disks from the same manufacturer; but Sun would not support them although it ship them in its storage appliances). This situation did not improve with T1000 and T2000 which for some stupid reason are limited to 80G drives. Sun Opteron servers hardware compatibility is much better and close to Dell and HP servers in this respect, with the hardware compatibility list that is pretty wide and constantly growing. So if ability to work with the large variety of harddrive is important (mail servers) I would recommend using NAS (this is a modern trend -- hardware subsystems should be separate from the server -- that opens the possibility of using blade servers in large enterprises. Otherwise you might be better off skipping T1000 and T2000 in favor X4100: open source applications work equally well on both architectures.

Until late 2006 the Sun eCommerce site was very weak and level of customarization and the variety of components (vital for entry level servers) that Dell provides is far ahead of Sun. Much like clueless advocacy discredits Linux, limited selection of hardware components (especially hard drives options) user unfriendliness, and interface design blunders of Sun eCommerce site disqualifies Sun from many potential low-end server deals. This happens because without clear understanding of the price and options available (based on quick Web site quote) it is difficult to judge the cost effectiveness of Sun-based solutions, unless you already has experience with a particular configuration. Also for low level sensors NAS appliance might make sense for email server (but it increases the costs making the Sun solution less competitive) but not much else. Requesting each time a quote from Sun reseller or several Sun reseller is not acceptable way of dealing with this problem for low end servers.

In Aug, 2006 I tried to obtain information about the possibility of installing 500G SATA drive into T1000 server (I have had a specific need for larger drive). This was not a pleasant experience to say the least. First Sun rep on direct support line incorrectly told me that installing such a drive will void server hardware warranty. I actually doubt that Sun-branded 500G SATA drive (probably Seagate in disguise) produces more heat then 80G drive. Knowing that nothing is perfect I double checked this fact with Sun distributor but to find "partial truth" took me two days and half-dozen dozen of emails. The final advice that I got was as following:

... it will not be supported by Sun. It will NOT void his warranty on the server, but if there is a problem with the server, they will likely blame it on the hard drive. The 500gb hard drive is not recommended, but it should work inside the server.

For comparison I checked Dell Small business site and in ten minutes had a configuration of PowerEdge 1950 with slightly better (2*300G SAS disks) configuration and 3 year support for both hardware and software (Dual Core Intel® Xeon® 5060, 2x2MB Cache, 3.20GHz, 1066MHz, 4GB 533MHz (4x1GB), Dual Ranked DIMMs, Red Hat Enterprise Linux ES 4, 3YR Red Hat Network Subscription, two 300G SAS disks, additional Intel® PRO 1000PT Dual Port Server Adapter, Dell Remote Access Card, 5th Generation for PowerEdge Remote Management, 3Yr SILVER ENTERPRISE SUPPORT: 7x24 HW/SW, 24X IDE CD-RW/DVD ROM Drive, etc) for approximately the same price ($5500).

What is interesting that Sun is almost as restrictive with their X4100 servers as with T1000 as for hardware drives installation. That is completely reasonable for midrange servers as SANs are the way to go, but
I think that this is questionable decision for low end.

As of Jan 2007 Sun Web commerce site was significantly improved and now capabilities and flexibility match Dell's. It even has some more advanced elements in design like thumbnails of components you select, the feature which helps to prevent clerical errors.

Mouse support is linux's weak spot and it hurts in enterprise environment. It often freezes on KVM switches (even expensive corporate-level switches with full emulation like Avocent).

CPU Efficiency

Until mid 2006 release of Intel dual core CPUs (51xxx series) Opteron was the king of the hill in raw CPU efficiency. After that leader changed and Intel is the leader. Currently Sun does not have Intel support for Solaris on the level it has for Opteron so Solaris suffers. But this changed in Jan 2007 with the agreement between sun and Intel. the results probably will be visible at the end of 2007.

Anyway this race for faster CPUs between Intel and AMD goes for a long time with one company topping other for a couple of years and then situation reverses. So whether Sun wishes or not it needs to support both CPU family for Solaris to match linux.

But in reality the speed of memory dominates in any non-trivial benchmarks. So differences obtainable with higher CPU speeds due to better caching might be non-reproducible outside the typical benchmarks with typical for them ultra high level of tuning of the hardware to extract from it as much horsepower as possible.

While memory speeds have increased from 4Mhz to 1.3GHz (approximately 300 times) since the first x86 CPUs were introduced 25 years ago, internal processor speeds have gone up by a at least twice higher rate. At some points of Intel-compatible PC history the difference between CPU speed and memory speed were as high as 10 times (for example, 1.3Ghz CPUs with 133 MHz memory).

This difference between CPU speed and memory speed partially explains why real life applications can behave worse or even much worse then the benchmark. And the lowest point is always more correlated with the speed of memory then the speed of CPU (it is possible to write a program that defeats all cache-based and CPU-based optimizations).

With the speed of memory limited to approximately 1.3 GHz (667 MHz memory chips) and any CPUs speed, say, over 2.66 GHz usually have marginal influence on total speed of execution of memory intensive applications unless those applications have "strong locality" property (many mathematical applications has this property, but database applications usually don't) and size of cache is large enough to accommodate "working set" of memory locations (that also means that CPU models with larger cache are usually superior to faster CPU models with smaller cache). For example Xeon 51xx processors have 4M L2 cache while Opteron CPUs have 1M of L2 cache per core.

It is true that multilevel cache architecture (L1+L2 ) somewhat smooth the discrepancy between speed of CPU and the speed of memory but there is no free lunch and penalties for cache miss are substantial. That's why throughput of CPU-cache tandem so much depends on the locality of references.

Quality of memory controller (processor bus) is another factor. The processor bus is tied to the external processor pin connections and can transfer 1 bit of data per data line every cycle. Most modern processors transfer 64 bits (8 bytes) of data at a time. The bus transfer rate, often called the bandwidth, represents the maximum speed at which data can move.

The Opteron uses an integrated dual-channel DDR memory controller rather than the traditional North Bridge design. Most specialists agree that Opteron has a design edge here because of the multiple data transfers across the memory bus and lower capabilities for sustained memory throughput, in current Intel architectures [VTL Under Fire]. Opteron also uses three HyperTransport tunnels to carry traffic between the processor and the chipset. See AMD comparison which is naturally biased toward Opteron, but still provides realistic picture of the differences.

Other things equal it is always better to buy servers with the highest possible speed of memory available from the vendor: saving on memory size and memory speed as probably the worst type of configuration mistakes people make. It is prudent to sacrifice speed of CPU to buy more memory and/or faster memory (and disks).

The clock frequency in the range of 2.2-2.6 GHz makes the Opteron is an interesting alternative for RISC CPU including Sun UltraSparc and IBM Power5. Some claims that it was Opteron that sunk Intel Itanics :-) The HyperTransport interconnection improves the efficiency of SMP-type clusters. A very large (over 11,000 processor) Opteron cluster with the HyperTransport facility, the Red Storm system, has been built at Sandia National Laboratory by Cray Inc.

Benchmarks

As both Solaris and linux run on Opteron they have close scores on most computational benchmarks. Large differences are visible only in more complex benchmarks, for example in transactional benchmarks.

As for computational benchmarks situation is not that good for UltraSparc. High end Fujitsu UltraSparc compatible CPUs are the only single core UltraSparc chips competitive with Opteron on SPECint2000/SPECfp2000 benchmarks, but they are more expensive then Opteron.

At the same time Solaris 10 on low end UltraSparc servers with new T1 chips from Sun are competitive on transactional benchmarks with high-end Opterons (T1 has 6 or 8 cores). We already published the results of SPECweb2005 which is one of the most relevant benchmarks for low end servers now we will provide some information about benchmark itself:

SPECweb2005 is the next-generation SPEC benchmark for evaluating the performance of World Wide Web Servers. As the successor to SPECweb99 and SPECweb99_SSL, SPECweb2005 continues the SPEC tradition of giving Web users the most objective and representative benchmark for measuring a system's ability to act as a web server. In response to rapidly advancing Web technology, the SPECweb2005 benchmark includes many sophisticated and state-of-the-art enhancements to meet the modern demands of Web users of today and tomorrow:

Measures simultaneous user sessions

Relevant dynamic content: PHP and JSP implementations included

Page images are requested using 2 parallel HTTP connections

Multiple, standardised workloads: Banking (HTTPS), E-commerce (HTTP and HTTPS), and Support (HTTP); agreed to by major players in the WWW market

Simulates browser caching effects by using If-Modified-Since requests

File accesses more closely matching today's real-world web server access patterns

Full disclosures available on this web site

Stable implementation with no incomparable versions

Java-based client for cleaner, more portable code

As Paul Merphy noted in his paper "The megahertz myth and the UltraSPARC T1"[Merphy2006]:

There are also far more trivial consequences, including one I happen to be interested in right now. One of the things that's going on in this benchmark is that the Xeons spend a lot of their time just waiting for memory - and, in fact, the faster they go, the higher the percentage of time they spend doing no-ops. Think about this in terms of marketing claims about gigahertz and you can see that Sun's occasional description of the T1 as a 9.6Ghz machine (because 8 x 1.2 = 9.6) can understate reality by a factor of at least three - since it takes a minimum of 27 Xeon Ghz to match it - and the Xeon's are running without the JVM overhead.

...Sun could reasonably claim a kind of "cycle equivalence" on web services for its T2000 at about 27Ghz in comparison to Xeon (and nearly 44Ghz in comparison to the Pentium IV). Now "Cycle equivalence" is not a concept I'd like to defend in the abstract, but I think it gives a rough and ready indicator of just how revolutionary this whole concurrent multi-threading idea really is - and that 1.75:1 ratio provides a good estimate for the low end of the range on Intel vs. PPC cycle equivalence on work done.

Despite this impressive result, generally, for low-end servers it is almost impossible to beat the advantage of running on the most popular hardware platform in existence. With Solaris 10 and T1 CPU Sun got a combination that is currently competitive in price/performance with Opteron-based servers but it's unclear for how long as dual-core Opteron CPUs are now reality. Recently Intel has begun shipping 'Woodcrest', its dual-core, 65nm Xeon processor based on its next-generation 'Core' architecture. Intel also released a low power - 40W - version of Woodcrest that eliminated power consumption advantages of T1 on low end. It will release four-core chips in the first quarter of 2007 and plans to start producing eight-core processors when it moves to a 45nm manufacturing process in 2008.

Generally this situation looks like uphill battle. And not only for Sun, but also for IBM with its "last hurrah": recent Power5 push. Big Blue is spending billions to upgrade its chip plants -- and getting thrashed by rivals that were manufacturing CPUs at much higher volumes and offering bargain-basement prices. It looks like IBM with its Power 5 push got in exactly the same position as Sun, but without benefits of strong partnership with AMD. It's one thing to beat in price/performance ratio older Sun V210/V240 servers with 1.34GHz UltraSparc IIIi CPUs. But to play a catch game with AMD and Intel relentlessly improving their lines of CPUs and putting billions and billions in research and manufacturing is completely another story and here IBM looks very vulnerable. Here is the relevant IBM press release reflecting their attempt to preserve their share of low end servers at the expense of Sun:

"With the introduction of the p5-510, IBM is taking advanced AIX 5L and POWER5 technology and making them available in an aggressively priced entry-level system," said Karl Freund, vice president, IBM eServer pSeries®. "IBM's commitment to capture the growing opportunity in high-volume and channel-based UNIX server sales takes a great leap forward with the introduction of this innovative server."

The system's slim form factor, innovative design features and system flexibility combine to provide a system unmatched by HP and Sun -- with superior performance to comparable offerings:
--  Performs 1.79X better than the Sun Fire V240 on integer(1)
--  Performs 1.46X better than the Sun Fire V240 on floating-point(1)
--  Performs 2.3X better than the Sun Fire V250 on NFS file serving(2)
--  Performs 1.1X better than the HP rx2600 in Java Virtual Machine
    serving(3)

Power Efficiency

Please note the power dissipation issue now become a very important both on both low end and mid range servers. Please look for the same Anandtech's benchmark that we already discussed above but from the point of view of power dissipation:

Specweb2005

"SPECweb2005 emulates users sending browser requests over broadband Internet connections to a web server," De Gelas explains. "It provides three new workloads: a banking site (HTTPS), an e-commerce site (HTTP/HTTPS mix), and a support site (HTTP). Dynamic content is implemented in PHP and JSP."

Sun Fire T2000: Processors: 1x 1.2GHz UltraSparc T1; Power Dissipation CPUs: 72-79 W; # of Cores: 8; # of Active Threads: 32 Score: 14,001; % Score: 289%

IBM p5 550: Processors: 2x 1.9GHz POWER5+; Power Dissipation CPUs: 320-360 W; # of Cores: 4; # of Active Threads: 8; Score: 7,881; % Score: 162%

IBM xSeries 346: Processors: 2x 3.8GHz Xeon; Power Dissipation CPUs: 220-260 W; # of Cores: 4; # of Active Threads: 4; Score: 4,348; % Score: 90%

Dell 2850: Processors: 2x 2.8GHz DC Xeon; Power Dissipation CPUs: 260-300 W; # of Cores: 4; # of Active Threads: 8 Score: 4,850 % Score: 100%

Operational costs of large commercial data centers are almost directly proportional to how much power is consumed by the servers. Unlike small "mom and pop shops" the challenge for large datacenters is not only reducing unit hardware cost, but also reducing energy consumption. Intel servers (and IBM Power5 servers) are relatively inefficient from this point of view. Opteron and especially T1 CPUs are relatively more energy efficient. In low-end servers, the processor typically accounts for 50% to 60% of the total consumption (with power supply as the next biggest consumer of energy).

Few analysts who discussed "linux vs Solaris" problem noticed that the problem with replacing one powerful server with several smaller one (path often advocated by linux enthusiasts) creates energy consumption problem as efficiency of converters from alternative current to DC in small servers and mid range servers is pretty low. DC power supplies typically have only 70% efficiency (50% on low loads). So additional costs of air-conditioning of a tropical landscape of a typical server room with servers instead of palms are substantial. It also limits expandability and creates the necessity of more expensive and more powerful power feeds. I know one company that instead of moving datacenter from one building to another when it moved to a small headquarters retrofitted additional (former factory) building as cost of providing additional power feed and air-conditioning in smaller building would eat all savings from the move from larger headquarters to smaller.

With relentless growth of the number of servers in the datacenter, energy efficiency is especially acute problem for those with datacenters that are located in high rise buildings. The latter usually have limitations on both power and air conditioning expandability. A typical Intel 51xx dual-core processor server now matches in power consumption both Opteron and UltraSparc. Previous Sun energy-consumption advantage when Intel consumed around 300 watts, which requires additional 50% (~150) watts of power for air conditioning disappeared in mid 2006.

In any case due to low efficiency of power supplies using low end servers you need to pay approximately 0.3 KWH per server. With a hundred servers (typical large datacenter figure) that creates power consumption of 30 KWH. Assuming the cost of generation $0.05 per KWH and the cost of delivery $0.04 per KWH the annual cost of energy per server is approximately $250, the figure which represents a significant share of maintenance costs.

Energy efficiency presents a formidable problem for Goggle which is a poster child of using low end linux server farms. Over the last three generations of Google's computing infrastructure, performance has nearly doubled. But because performance per watt remained nearly unchanged, that means electricity consumption has also almost doubled. That is a typical problem for any firm that tries to building massive server clusters on commodity hardware. The hardware is cheap, but it is designed to solve a different problem, powering a box in a desktop environment. As Google vice president of operations Urs Hoelzle noted "'Over four years, the power costs of running a PC can add up to half of the hardware cost". If we assume that price of electricity can double like the price of gas did, the four-year cost of a server's electricity bill will soon be larger than the $3,000 initial price of a typical low-end server with x86 processors.

Google try to fight the problem using more efficient (and more expensive) power supplies and to work with component makers to accelerate the time-to-market of more efficient devices, such as motherboards with a smaller number of DC voltage inputs [TG Daily]. "Saving power is still the name of the game, even to the extent that we shut off the lights in them when no-one is there" noted Hoelzle.

Comparative security and virtualization capabilities

I would like to stress it again that there are significant security advantages of using different, less popular hardware and different target compiler for the security of all major open source applications. The result is the same open source application with the same vulnerabilities are more secure on Solaris and thus requires less frequent patching. An argument that this is security via obscurity is simply naive (or crooked): obscurity is an essential part of real-world security.

Open source applications like bind and Sendmail compiled with Sun Studio 11 complier are definitely less susceptible to the "off the shelf" exploits which are mainly designed for GCC-compiled binaries. Here open source applications provide a significant security edge as an intelligent customer can chose the compiler himself and this decision become an additional variable (secret key) that needs to be accounted for in exploit writing.

Solaris 10 also has an excellent implementation of RBAC and light-weight VM (zones). Both add to security in a major way.

Linux generally badly miss virtualization bandwagon. And those capabilities are very important on low end: many applications that benefit from a separate server actually does not consume much resources. Also development and staging servers can usually be virtualized without any problems. Currently only linux distributions with Xen support can complete with Solaris 10 in this area. As of June 2006 none of enterprise linux distributions is available with XEN. That means that Solaris beats linux in this area by providing this capability for more then a year. In other works zones are probably mature enough today to use in enterprise environment. XEN based Linux distributions will achieve this status somewhere in 2007 or even 2008. In other words, taking into account the conservatism of enterprise customers Solaris might well enjoy a competitive advantage for at least a year, when RedHat 5 and similar enterprise distributions reach the stability that justifies their deployment. Only suicidal enterprise managers deploy version of Unixes that are less that a year old and in case of Linux distribution two years lag is often a better deal. And it does not matter one bit that that enthusiasts can use Xen now, it is still not enterprise ready and probably will not be until at least late 2007.

That means that Solaris 10 can compete for a bigger share in enterprise web hosting space, the area where linux made the most significant advances. In web hosting environment when each web server is running in its own zone, the attack using an available exploit for the application or script running of this web server leads only to the compromise of this particular zone environment that contain nothing but one instance of a web server. All other webservers on the same server (running in different zones) are still immune from defacing and similar threats. Only FreeBSD currently provides a similar and mature enough mechanism of virtualization for this purpose. Whether Sun will be able to realize this temporary advantage remains to be seen.

If you working is a sophisticated (or mixed with Windows environment) then the support of ACLs is essential and here Solaris is much better then linux (at least as of spring 2006).

In it important to note that BSD line of OSes (this list includes FreeBSD, OpenBSD, NetBSD) have prided themselves on supplying professional, reliable, secure and high performance low end servers, especially for ISPs. They are more democratic then enterprise linux with really free patches and nice portage system. And they really shine on single CPU servers despite all PR fog around linux. Linux still have a home field advantage as it is the development platform for many (but not all) important for enterprises and ISPs open source applications (ssh, bind and sendmail are actually "BSD-friendly" applications, to name just a few).

Moreover Linux development now is financed mainly by IBM, Oracle, Intel and HP (Red Hat is more a mutual fund that is kept alive by forces that are outside its own control, then a regular commercial OS vendor; Novell like Sun has issues with profitability). That means that Linux kernel developers are covering too large territory now and might partially lose their focus on low end hardware. In this sense linux is on the road of becoming just average everywhere including low end. FreeBSD 4.x is a rock solid offering in low end space for major enterprise applications, the server OS that can be put down only using AK47 :-). Linux is just good enough. But FreeBSD is suffering from the same problem of yet another OS that needs to be added to the enterprise mix. So all things equal Solaris is more "politically correct" choice for large enterprise, even if we all agree that undeniable technical merits of BSD OSes fully justify a second look on the problem. Unfortunately software world is very convoluted and technically best offerings seldom enjoy most market success. And that's true not only for operating systems.

It's important to note that each of the three BSD flavor (FreeBSD, NetBSD and OpenBSD) have a different emphasis (FreeBSD emphasize PC compatibility, NetBSD -- portability and OpenBSD -- security and simplicity of codebase). All of them are quite competitive to linux on low end and in several aspects are architecturally superior (especially FreeBSD, which introduced light-weight VM called jails in 1999), which is amazing achievement taking into account a tiny fraction of resources used for development in comparison with Linux. Even if we assume one year delay for the stabilization of jails, BSD users were able to enjoy better security for six more years then Linux users.

Administration-friendliness and the ability to work with OS images and simplify system-to-system migration (coping images between servers., etc).

As for administration friendliness linux leaves Solaris in the dust. In Solaris standard configuration they are not able even to provide a decent .profile script for root. And in many companies I see servers used in such way for years with administrator starting with commands like 'stty erase' and using prompt '#" all the day. Actually once I have on the floor Sun's certified instructor as a consultant and what surprised me that while configuring servers he never installed any .profile although this approach naturally led to tremendous amount of extra typing -- so if Sun's instructors work this way, what can you say about a typical Sun administrator :-).

Similar thing happened to virtualization. Paradoxically the key advantage of virtualization proved to be not its potential ability to consolidate many small servers into one large one, but easing server-to server-migration path and creating cheap "para-cluster" solutions on one physical servers (and Sun servers are really very reliable). Zones are the greatest Solaris advantage in virtualization area, the advantage that currently linux cannot match. Imitation of this capability using full hardware virtualization with VMware is prohibitively expensive and has twice higher overhead (zones overhead is within 10%, VMware is over 20%).

Xen, more powerful and elegant para-virtualization engine for X86 architecture developed at Cambridge, is still an unproven technology that will find its way into enterprise not earlier then 2007. It provides capability similar to VMware but with overhead more similar to Solaris Zones overhead. The key advantage of VMware is dramatic simplification of system-to-system migration path which is a very important for low end servers. Xen has similar capabilities: copy the boot drive using dd command to another server, point Xen at that file, and boot the VM. It also provides the capabilities for migrating a running VM from one physical server to another "on the fly", the key attraction of VMware Virtual Center. In both cases most hardware differences are resolved on virtualization level.

As for images Solaris has an excellent tool called flash archives. Zones also provide rudimentary "image swapping" capabilities. Also like linux vendors Sun is working on integrating Xen in Solaris so it will be the second virtualization capability available for the OS. Sun will release working Xen support code in the second half of 2006. This code will give OpenSolaris the ability to run on Xen as a Dom0 (host system), with support for 32-bit and 64-bit guests (DomU). Sun plans to integrate XEN into regular releases of Solaris 10 in 2007.

Running LAMP Stack

Linux has home field advantage for LAMP stack. At the same time the lion share of testing and almost all documentation for major open source applications including databases is currently limited to 32-bit packages. Due to this reason, sometimes Sun supplies 32-bit packages of LAMP stack even on 64-bit UltraSparc. For example, in CSQamp package (a precompiled with Studio 11 LAMP stack) which is a part of Cool Tools set of optimized for T1 architecture packages, PHP and MySQL are included as 32-bit applications (CSQamp package will work with the CSQmysql -- 64-bit MySQL database server):

CSQamp. This package includes Apache HTTP Server 2.0.58, MySQL 5.0.22 and PHP 5.1.4 built to work together. Apache httpd is built with MPM pre-fork and modules to support PHP, SSL and Perl. PHP has support for MySQL. Note, to work with PHP, MySQL included in this package is a 32-bit version. For the database server, we recommend you install the CSQmysql package which is a 64-bit version, allowing the use of larger caches to deliver improved performance for large databases. Note that the CSQamp package will work with the CSQmysql database server.

Currently only Solaris on Intel is compartative in speed with Linux running LAMP stack. T2000 is also a contender but it is more expensive.

Contents

Copyright © 1996-2008 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. Submit comments This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

Standard disclaimer: The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Created Jan 2, 2005. Last modified: February 28, 2008