Notes on five different types of virtualization

The latest Holy Grail of large corporate IT is server consolidation. In an attempt to lower the costs of IT infrastructure many companies are looking to achieve this via server virtualization. The idea is to consolidate small and not very loaded servers into fewer and larger as well as more heavily-loaded physical servers. This can bring up a whole new set of complications, however, as there is no free lunch and if overdone the price should be paid. But in moderation this new trend that can be called conversion of IT environment into set of Virtual Machines can be quite beneficial and opens some additional, non-foreseen avenues of savings.

Also if done intelligently virtualization can probably squeeze the number of the servers in a typical datacenter 30-50% and that also lead to some modest maintenance cost savings as well as electricity and air-conditioning related savings (low end servers as very wasteful as for electricity and add considerable to air-conditioning costs as their power supplies are very inefficient).

Saving on hardware is more questionable as low level servers represent the most competitive segment of server market with profit margins squeezed to minimum; the margins are generally much larger of mid-range and high end servers. In other words margins on midrange servers and high-end servers work against virtualization.

At the same time the heavy reliance on virtualized servers for production applications, as well as the task of managing and provisioning them, are fairly new areas in the "new brave" virtualized IT world and both need special software solutions. That increases the importance of Tivoli and other ESM applications. Virtualization has changed configuration management, capacity management, provisioning, patch management, back-ups, and software licensing. It is inherently favorable toward open source software and OS solutions. It also opens a lot of new possibilities in saving time on administration, electricity savings as well as makes some impressive feats like dynamic migration of a virtual instance from one (more loaded) physical server to another (less loaded) possible.

Major Types of Virtualization

Super-heavy-weight

This is hardware domain-based virtualization that is used only on high-end servers. Domain can, essentially, be called "blades with common memory and I/O devices". Those "blades on steroids" are probably the closest thing on getting more power from a singe server without related sacrifices in CPU, memory access and I/O speed, sacrifices that are typical for all other virtualization solutions. Of course there is no free lunch and you need to pay for such luxury. Sun is the most prominent vendor of such servers (mainframe class servers like E15K, etc are all hardware domain-based).

Access to memory of other domains is slower then to local memory so those systems are closer to NUMA.

Heavy-weight

IBM calls it LPARs and is currently the king of the hill in this area as it pioneered this class of VM machines in late 60 with the release of famous VM/CMS. Until recently Power5 based server with AIX were the most battle-tested and reliable virtualized environments based on heavy-weight virtualization concept.

Still the most popular implementation of this concept currently is VMware and recently it was greatly helped by Intel and ADM who incorporated virtualization extensions in their CPUs. VMware can run Linux (Red Hat and Suse), Solaris and Windows virtual instances on one physical server and as such is the most versatile solution on this category although IBM Power5 servers still enjoy the advantages of hardware designed with the virtualization in mind.

But with Intel quad CPUs available you can have pretty impressive CPU power on a single Intel server and that makes VMware more important as rising tide of Intel server power (as well as AMD competitive efforts to match Intel after its introduction of 5xxx series of CPUs -- Duo and Quattro) lifts all boats.

On newer Intel and AMD CPUs Xen also can run unmodified OS instances making it another heavy-weight virtualization platform.

Sun calls heavy-weight virtual partitions "logical domains"(LDOM) and until recently preferred hardware-base domains to logical domains. But this stance is changing with the introduction of LDOMs on Sun's T1000 and T2000 Sun Fire servers. The first is low end server and as such is questionable platform for hevy-weight virtualization, the second is actually something in between low end and middle-weight server and can run at least two or may be three virtual partitions with substantial simultaneous loads. I am not sure what is the speed of memory for T2000 but I doubt that it is 1.33GHz. It is probably lower so in this area T2000 is inferior to newer Intel 5xxxx based servers.

Customers that use Solaris 10 11/06 can turn to new hardware being shipped in January or a firmware update on older boxes in order to get LDOMs working on their UltraSPARC T1-based servers. Sun will support up to 32 operating systems per server with the virtualization technology. About differences with LPARs see Rolf M Dietze blog. Among other things he came to the following conclusions:

CPU vendors now are paying huge attention to virtualization. All new CPUs are usually "virtualization-friendly" and contain instructions and hardware capabilities that make heavy-weight virtualization more efficient. Intel 5xxx series (Duo and Quattro CPUs) is an example of this class of CPUs among Intel-compatible CPUs. IBM P5 and Sun UltraSparc T1 are examples among RISK CPUs.

The main advantage of heavy-weight virtualization is almost complete isolation of instances. You cannot achieve this with any other type of virtualization and here only blades can compete.

Disadvantages are connected with the fact that both CPUs, memory and I/O are shared and you will never get the same speed on high workloads as in case of several standalone servers each with corresponding fraction of CPUs and memory and the same set of applications. Especially problematic is sharing of memory as it might well become a bottleneck before CPU. Each virtual instance of OS loads pages independently of the other and compete for memory bandwidth. So if, for example, two virtual instances are simultaneously active and for example are loading modules or data from the disk each can enjoy probably only 2/3 of the memory bandwidth (accesses to memory are randomly spread in time so sum should probably be greater then 100%) in comparison with a standalone system. In other words you lose approximately 1/3 of memory bandwidth by jumping into virtualization bandwagon. That's why heavy-weight virtualization behaves bad on memory intensive applications. Users of IBM Power5 servers and AIX 5.3 (probably are the best and the most widely used commercial heavyweight virtualization platform) know that all too well.

As memory channel is shared between all virtual instances heavy-weight virtualization is best suited for Web and e-commerce workloads.

That probably means that other things equal you should strive for the fastest memory and the best designed memory bus (Opteron has better memory bus then Intel CPUs). That's probably why new 5xxx series of CPU that support 1.3Ghz memory significantly improves the performance of the VMware (see Dell paper about the improvements achieved).

Both memory speed and CPU power can became bottlenecks as they are shared between all active virtual instances of OS. The presence of a full non-modified version of an OS for each partition introduces significant drag on resources (both memory and CPU-wise). I/O load can be diminished by using a separate controller for OS partitions and multiple controllers on the server. Still in some deep sense heavy-weight partitioning is inefficient and will always waist significant part of server resources.

In IBM Power-based server world "binary" deployments when a single server host two application in two different LPARs are common pretty popular and behave reasonably well. Taken to the extreme this approach guarantee 50% reduction of all physical servers. Larger number of applications on a single server are possible, but more tricky: such virtual server need careful planning as it faces memory bottleneck and CPU power bottleneck, especially painful if "rush hours" are the same for both applications. We need to understand that a larger server with two CPU and 8G of memory split equally between two partitions will always be less efficient then two several servers with one CPU and 4G of memory if speed of memory and speed of CPU are equal. As an alterative to heavy-weight partitioning see blade servers discussion below.

This approach is important for running legacy applications like application for SunOS 4.x as well as few linux applications that makes sense to run of Linux instead of Solaris but that need to run on the same server.

Medium-weight (para-virtualization)

Para-virtualization is a variant of native virtualization, where the VM (hypervisor) emulates only part of hardware and provides a special API requiring OS modifications. The most popular representative of this approach is Xen :

Therefore only specially modified for Xen versions on OS can run in virtual mode. Work on Xen has been supported by UK EPSRC grant GR/S01894, Intel Research, HP Labs and Microsoft Research (Yes, despite naive Linux zealots wining Microsoft did contributed code to Linux ;-). Other things equal it provides higher speed and less overhead then native virtualization. NetBSD was the first to implement Xen. Currently the key platform for Xen is linux with Novell supporting it in production version of Suse. Red Hat does not support it in RHEL 4 but is expected to support in RHEL 5 somewhere in 2007. Sun Solaris 10 for x86 also have Xen support (currently in beta with production version in early 2007).

Xen is now sold commercially by IBM; Sun will have Xen compatible version of Solaris in mid 2007. SPARC has separate implementation that will be released in late 2007 and two implementations re expected to merge in the future).

The main advantage of Xen is that it supports live relocation capability. It is also more cost effective solution the VMware that is definitly overpriced.

The main problem is that like with any para-virtualization solution the OS needs to be modified to be aware of the environment it is running and pass control to hypervisor in case of executing all privileged instructions. Therefore is not suitable for running legacy OSes.

Para-virtualization improves speed in comparison with heavy-weight virtualization, but does little beyond that. It is unclear how much faster is para-virtualized instance of OS in comparison with heavy-weight virtualization on "virtualization-friendly" CPUs. Xen page claims that:

It's unclear was this difference measure of old Intel CPU or new 5xxx series that support virtualization extensions.

I would like to stress it again that the level of modification OS is very basic and duplicate function like virtual memory management are not factored out. Therefore all the redundant processing typical for heavy-weight virualization in present in para-virtualization environment.

Note: Xen 2.0 had the initial support for para-virtualization, meaning that guest OSes would have to be modified to run on top of the hypervisor. Xen 3.0 and above support both para-virtualization and full (heavy-weight) virtualization to leverage the inbuilt hardware support built into the Intel-VT-x and AMD pacifica processors. According to XenSource Products - Xen 3.0 page:

Light-weight virtualization

This type of virtualization was pioneered in Free BCD (jails) and was further developed by Sun and introduced in Solaris 10 as concept of Zones. There are various experimental add-ons of this type for Linux but none got prominence.

In Solaris 10 11/06, the next build of the operating system that will be released at the end of November, admins will be able to clone a Zone as well as relocate it to another box, through a feature called Attach/Detach. Also now it is possible to run Linux applications in zones on X86 servers (branded zones). The key advantage is that you have a single instance of OS so the price that you paid in case of heavy-weight virtualization is waived. That means that light-weight virtualization is the most efficient resources-wise. It also has great security value. Memory can become a bottleneck here as all memory accesses are channeled via a single controller.

IBM's "lightweight" product would be "Workload manager" for AIX which is an older (2001 ???)and less elegant technology then BSD Jails and Solaris zones:

IBM lightweight virtualization operates under a different paradigm with the most close thing to zone being a "class". The system administrator (root) can delegate the administration of the subclasses of each superclass to a superclass administrator (a non-root user). Unlike zones classes can be nested:

Blades

Blade servers are an increasingly important part of the enterprise datacenters, with consistent double-digit growth easily outpacing the overall server market. IDC estimated that 500,000 blade servers were sold in 2005, or 7% of the total market, with customers spending $2.1 billion.

While blades are not virtualization in pure technical sense, the rack with blades (bladesystem) possesses some additional management capabilities that are not present in stand-alone U1 servers and in modern versions usually have shared I/O channel to NAS. Still they can be viewed as "hardware factorization" approach to server construction which is not that different from virtualization. The first shot in this direction is the new generation of bladesystems like IBM BladeCenter H system has offered I/O virtualization since February, 2006 and HP BladeSystem c-Class. The latter offers better server management, virtualization and saves up to 30% power in comparison with rack mounted 1U servers with identical CPU and memory configurations. Sun also offers blades but it is a minor player in this area. It offers pretty interesting and innovative Sun Blade 8000 Modular System which target higher end that usual blade servers. Here is how Cnet described the key idea behind the server if the article Sun defends big blade server 'Size matters':

Sun co-founder Andy Bechtolsheim, the company's top x86 server designer and a respected computer engineer, shed light on his technical reasoning for the move.

"It's not that our blade is too large. It's that the others are too small," he said.

Today's dual-core processors will be followed by models with four, eight and 16 cores, Bechtolsheim said. "There are two megatrends in servers: miniaturization and multicore--quad-core, octo-core, hexadeci-core. You definitely want bigger blades with more memory and more input-output."

When blade server leaders IBM and HP introduced their second-generation blade chassis earlier this year, both chose larger products. IBM's grew 3.5 inches taller, while HP's grew 7 inches taller. But opinions vary on whether Bechtolsheim's prediction of even larger systems will come true.

"You're going to have bigger chassis," said IDC analyst John Humphries, because blade server applications are expanding from lower-end tasks such as e-mail to higher-end tasks such as databases. On the more cautious side is Illuminata analyst Gordon Haff, who said that with IBM and HP just at the beginning of a new blade chassis generation, "I don't see them rushing to add additional chassis any time soon."

Business reasons as well as technology reasons led Sun to re-enter the blade server arena with big blades rather than more conventional smaller models that sell in higher volumes, said the Santa Clara, Calif.-based company's top server executive, John Fowler. "We believe there is a market for a high-end capabilities. And sometimes you go to where the competition isn't," Fowler said.

As a result of such factorization more and more functions move to the blade enclosure. As a result power consumption improves dramatically as blades typically use low power dissipating CPUs and all blades typically share the same power supply that in case of full or nearly full rack permits power supply to work with much greater power efficiency (twice of more efficient then on a typical server). That cuts air conditioning costs too. also newer blades monitor air flow and adjust fans accordingly. As a result energy bill can be half of the same amount of U1 servers.

Blades generally solves the problem of lack of CPU power typical for most types of virtualization except domain-based, and with the current price of memory it solves the memory latency problem. Think about them are predefined partitions with fixed amount of CPU and memory. Dynamic swap of images between blades is possible. Some I/O can be local and with high speed solid drives very reliable and fast. That permits offloading OS-related IO from application related I/O.

Major vendors support

Conclusions

There is no free lunch and virtualization is not panacea. It increases the complexity of environment and put severe stress of a single server that host multiple instances on virtual machines. Failure of this server lead to failure of all instances.

Therefore the natural habitat of virtualization is development, test and stage servers as well as almost idle servers that servers various enterprise consoles and similar low CPU intensive applications (Web servers and e-commerce servers).

At the same time virtualization opens new capabilities and sometimes it make sense to run a single instance of virtual machine on the server to get such advantages as on the fly relocation of instances, virtual images manipulation capabilities, etc. With technologies like Xen that claims less then 5% overhead that approach becomes feasible. "Binary servers" -- servers that host just two applications also look very promising.

Migration of rack-mounted servers to blade servers is the most safe approach to server consolidation. Managers without experince of work in partitioned environment shouldn’t underestimate what their administrators need to learn and the set of new problems that cirtualization creates One good advice is "Make sure you put the training dollars in."

There are also other problems. A lot of software vendors won’t certify applications as virtual environment compatible, for example VMware compatible. In such cases running the application in virtual environment means that you need to assume the risks and cannot count on vendor tech support to resolve your issues.

All-in all virtualization is mainly played now inIt make sense to proceed slowly testing the water before jumping in. Those that have adopted virtualization have, on average, only about 20% of their environment virtualized, according to IDC. VMware pricing structure is a little bit ridiculous and nullifies hardware savings, if any. Their maintenance costs are even worse. That means that alternative solutions like Xen3 or Microsoft should be considered on Intel side and IBM and Sun on Unix side. As vendor consolidation is ahead if you don’t have a clear benefit from virtualization today, you can wait or limit yourself to "sure bets" like development, testing and staging servers. The next version of Windows Server will put serious pressure on VMware in a year or so. Xen is also making progress with IBM support behind it. With those competitive pressures, VMware could become significantly less expensive in the future.

VMs are also touted as a solution to the computer security problem. It's pretty obvious that they can improve security. After all, if you're running your browser on one VM and your mailer on another, a security failure by one shouldn't affect the other. If one virtual machine is compromised you can just discard it and create an fresh image. There is some merit to that argument, and in many situations it's a good configuration to use. But at the same time the transient nature of Virtual Machines introduces new security and compliance challenges not addressed by traditional systems management processes and tools. For example virtual images are more portable and possibility of stealing the whole OS images and running them on a different VM are very real. New security risks inherent in virtualized environments need to be understood and mitigated.

Notes on five different types of virtualization

Dr. Nikolai Bezroukov

Introduction

Major Types of Virtualization

Super-heavy-weight

Heavy-weight

Medium-weight (para-virtualization)

Light-weight virtualization

Blades

Major vendors support

Conclusions

Webliography