The Book of Xen Part 15

There are some excellent reasons to consider the idea. KVM and lguest are both easier to install and less invasive than Xen. They support strong virtualization with good performance.

However, KVM is, at the moment, less mature than Xen. It's not (yet) as fast, even with the kernel accelerator. Xen also supports paravirtualization, whereas KVM does not. Xen PV offers a handy way of migrating domUs and a good way of multiplexing virtual machines-that is, expanding to a two-level VM hierarchy. But honestly, we haven't got much experience with KVM, and we've got quite a lot with Xen.

Similarly, lguest is smaller, lighter, and easier to install than Xen, but it doesn't support features like SMP or PAE (though 64-bit kernels are in development). Lguest also doesn't yet support suspend, resume, or migration.

Nonetheless, right now it's difficult to say which is better-all of these technologies are out there being actively developed. If you are truly silly, you might even decide to use some combination, running Xen hypervisors under KVM, with paravirtualized domains under that. Or you might use Xen for now but keep your options open for future deployments, perhaps when HVM-capable hardware becomes more common. These technologies are interesting and worth watching, but we'll stick to our usual "wait and see" policy.

Indeed, Red Hat has opted to do exactly that, focusing its development efforts on a platform-independent interface layer, libvirt, allowing (we hope) for easy migration between virtualization options. See Chapter6 Chapter6 for more on libvirt and its a.s.sociated suite of management tools. for more on libvirt and its a.s.sociated suite of management tools.

Working with HVM Regardless of what sort of hardware virtualization you want to use, the first thing to do is check whether your machine supports HVM. To find out if you've got a supported processor, check /proc/cpuinfo /proc/cpuinfo. Processors that support VT-x will report vmx vmx in the flags field, while Pacifica-enabled processors report in the flags field, while Pacifica-enabled processors report svm svm.

Even if you have a supported processor, many manufacturers have HVM turned off in the BIOS by default. Check xm dmesg xm dmesg to ensure that the hypervisor has found and enabled HVM support correctly-it should report "(XEN) VMXON is done" for each CPU in the machine. If it doesn't, poke about in the BIOS for an option to turn on hardware virtualization. On our boards, it's under to ensure that the hypervisor has found and enabled HVM support correctly-it should report "(XEN) VMXON is done" for each CPU in the machine. If it doesn't, poke about in the BIOS for an option to turn on hardware virtualization. On our boards, it's under chipset chipset features features and called and called VT VT Technology Technology. Your machine may vary.

The hypervisor will also report capabilities under /sys /sys: #cat/sys/hypervisor/properties/capabilities xen-3.0-x86_32phvm-3.0-x86_32hvm-3.0-x86_32p In this case, the two "hvm" entries show that HVM is supported in both PAE and non-PAE modes.

NoteOne of the minor advantages of HVM is that it sidesteps the PAE-matching problem that we've been p.r.o.ne to harp on. You can run any mix of PAE and non-PAE kernels and hypervisors in HVM mode, although paravirtualized domains will still need to match PAE-ness, even on HVM-capable machines.

Creating an HVM Domain When you've got the hypervisor and domain 0 running on an HVM-capable machine, creating an HVM domain is much like creating any Xen guest.

Here's a sample HVM config file. (It's got a snippet of Python at the beginning to set the appropriate library directory, borrowed from the sample configs distributed with Xen.) importos,re arch=os.uname()[4]

ifre.search('64',arch): arch_libdir='lib64'

else: arch_libdir='lib'

device_model='/usr/'+arch_libdir+'/xen/bin/qemu-dm'

kernel="/usr/lib/xen/boot/hvmloader"builder='hvm'

memory=384 shadow_memory=16 name="florizel"

vcpus=1 vif=['type=ioemu,bridge=xenbr0']

disk=['file:/opt/florizel.img,hda,w','file:/root/slackware-12.0-install-dvd.iso,hdc:cdrom,r']

boot="cda"

sdl=0 vnc=1 stdvga=0 serial='pty'

Most of this is pretty standard stuff. It starts with a snippet of Python to choose the correct version of qemu-dm qemu-dm, then it launches into a standard domU config. The config file changes for HVM guests are approximately as follows: builder="HVM"

device_model="/usr/lib/xen/bin/qemu-dm"kernel="/usr/lib/xen/boot/hvmloader"NoteThere are other directives you can put in, but these are the ones that you can't leave out.

Breaking this down, the domain builder changes from the default to HVM, while the devices change from the standard Xen paravirtualized devices to the QEMU emulated devices. Finally, the kernel kernel line specifies an HVM loader that loads a kernel from within the HVM domain's filesystem, rather than the Linux kernel that would be specified in a PV configuration. line specifies an HVM loader that loads a kernel from within the HVM domain's filesystem, rather than the Linux kernel that would be specified in a PV configuration.

A DIGRESSION ON THE DOMAIN BUILDERUp until now, we've avoided discussing the domain builder, being content merely to gesture in its direction and note that it builds domains. For most purposes, it's enough to think of it as a.n.a.logous to the standard bootloader.However, it has a much more difficult and involved task than a normal bootloader. When a Xen domU starts, it comes up in a radically different environment from the traditional PC "real mode." Because an operating system's already loaded, the processor is already in protected mode with paging enabled. The job of the domain builder is to bring the new domain up to speed-to generate the mapping between domain-virtual and physical memory, load the VM kernel at the appropriate address, and install interrupt handlers.The default builder is linux, which builds a paravirtualized domain. (Usually Linux, but it'll work for most paravirtualized OSs.)The situation changes somewhat with HVM because the unmodified operating system isn't expecting to take off running on a fully booted machine. To keep the OS happy, the domain builder initializes a complete simulation of real mode, inserting hooks and installing an emulated BIOS into the appropriate regions of pseudophysical memory.Historically, the domain builder's been implementation-dependent, with a "vmx" builder for Intel and a similar "svm" builder for AMD. However, with the advent of HVM as an abstraction layer, the administrator can simply specify HVM and let Xen figure out the details.

We're already familiar with the kernel kernel line, of course. The line, of course. The device_model device_model line, however, is new. This option defines a users.p.a.ce program that handles mediation between real and virtual devices. As far as we know, line, however, is new. This option defines a users.p.a.ce program that handles mediation between real and virtual devices. As far as we know, qemu-dm qemu-dm is the only option, but there's no reason that other device emulators couldn't be written. is the only option, but there's no reason that other device emulators couldn't be written.

There are some other directives that are only used by HVM domains.

shadow_memory=16 boot="dca"

The shadow_memory shadow_memory directive specifies the amount of memory to use for shadow page tables. (Shadow page tables, of course, are the aforementioned copies of the tables that map process-virtual memory to physical memory.) Xen advises allocating at least 2KB per MB of domain memory, and "a few" MB per virtual CPU. Note that this memory is directive specifies the amount of memory to use for shadow page tables. (Shadow page tables, of course, are the aforementioned copies of the tables that map process-virtual memory to physical memory.) Xen advises allocating at least 2KB per MB of domain memory, and "a few" MB per virtual CPU. Note that this memory is in addition in addition to the domU's allocation specified in the memory line. to the domU's allocation specified in the memory line.

Finally we have the boot boot directive. The entire concept of boot order, of course, doesn't apply to standard Xen paravirtualized domains because the domain config file specifies either a kernel or a bootloader. However, because HVM emulates legacy BIOS functions, including the traditional bootstrap, Xen provides a mechanism to configure the boot order. directive. The entire concept of boot order, of course, doesn't apply to standard Xen paravirtualized domains because the domain config file specifies either a kernel or a bootloader. However, because HVM emulates legacy BIOS functions, including the traditional bootstrap, Xen provides a mechanism to configure the boot order.

In that vein, it's worth noting that one advantage of HVM is that it can effectively duplicate the QEMU install procedure we've already described, with Xen instead of QEMU. To recap, this allows you to install in a strongly part.i.tioned virtual machine, using the distro's own install tools, and end with a ready-to-run VM. We'll leave the details as an exercise to the reader, of course (we wouldn't want to take all the fun out of it).

Now that you've got the config file written, create the domain as usual: #xmcreateflorizel Usingconfigfile"./florizel".

Starteddomainflorizel #xmlist NameIDMem(MiB)VCPUsStateTime(s) Domain-0034582r-----5020.8 florizel63961r-----14.6 Interacting with an HVM Domain One of the first changes you might notice is that connecting to the console using xm -c xm -c doesn't work. The Xen console requires some infrastructure support to get working, which a PV-oblivious standard distro naturally doesn't have. doesn't work. The Xen console requires some infrastructure support to get working, which a PV-oblivious standard distro naturally doesn't have.

So, while the machine is booting, let's chat for a bit about how you actually log in to your shiny new domain.

As you're no doubt sick of us mentioning, HVM is founded on an idea of total hardware simulation total hardware simulation. This means that when the machine boots, it loads an emulated VGA BIOS, which gets drawn into a graphics window.

Xen knows about two targets for its emulated VGA graphics: VNC and SDL. VNC is the familiar and well-beloved network windowing system from AT&T, while SDL, Simple DirectMedia Layer SDL, Simple DirectMedia Layer, is better known as a lightweight hardware-accelerated graphics option.

We opted to stick with the VNC console for most of our Linux domUs to reap the benefits of network-centric computing.[78]

Now that the domain is created, use VNC to access its console: #vncviewer127.0.0.1 (Or use whatever the IP address of the Xen machine is.) If you have more than one domU using the VNC console, append a display number-for example, to access the console of the second domU: #vncviewer127.0.0.1:1 If the vncunused= vncunused= option in the config file is set, the domain will take the first available display number. If not, it'll take the display number that corresponds to its domain number. We tend to leave it set, but unset is fine too. option in the config file is set, the domain will take the first available display number. If not, it'll take the display number that corresponds to its domain number. We tend to leave it set, but unset is fine too.

Somewhat to our surprise, X11 worked quite well out of the box with the vesa vesa driver, taking over the VNC framebuffer console and providing a usable display without further configuration. driver, taking over the VNC framebuffer console and providing a usable display without further configuration.

Getting the Standard Xen Console to Work Now, logging in via the graphical console is an irritating bit of overhead and, we would argue, overkill for a server. Fortunately, you can circ.u.mvent this by using the standard Xen emulated serial console. First, make sure that your domU config file (e.g., /etc/xen/leontes /etc/xen/leontes) contains the following: serial='pty'

This directive tells QEMU (and therefore Xen) to pa.s.s serial output to the Xen console.

Now the bootloader needs to be told to pa.s.s its messages to the serial serial line. Add the following to line. Add the following to /boot/grub.conf /boot/grub.conf in the domU: in the domU: serial--unit=0--speed=115200--word=8--parity=no--stop=1serialconsole These two lines give GRUB some settings for the serial port and tell it to actually use the serial port as its console.

Next, set up the kernel to output its boot information via serial: t.i.tleCentOS(2.6.18-8.el5) root(hd0,0) kernel/vmlinuz-2.6.18-8.el5roroot=/dev/VolGroup00/LogVol00rhgbquietconsole=ttyS0 module/initrd-2.6.18-8.el5.img (If you're loading the Xen hypervisor in HVM mode-which is a perfectly reasonable thing to do-your menu.lst menu.lst file will look a bit different, of course.) file will look a bit different, of course.) Finally, edit inittab inittab to start a to start a getty getty on the emulated on the emulated serial serial line by adding a line like this: line by adding a line like this: s1:12345:resp.a.w.n:/sbin/agetty-LttyS09600vt100 Boot the machine, and you should be able to see messages and log in via both xm console xm console and VNC. and VNC.

[78] By which I mean, we didn't want to have to stand up and walk over to a test machine to access the console. By which I mean, we didn't want to have to stand up and walk over to a test machine to access the console.

HVM Devices If you poke around in your new HVM domain, you'll see devices like "QEMU harddisk," or "RealTek Ethernet controller." These take the place of the corresponding Xen devices, like xenblock xenblock or or xennet xennet. Examine this dmesg dmesg output, for example: output, for example: PIIX3:IDEcontrolleratPCIslot0000:00:01.1PIIX3:chipsetrevision0 PIIX3:not100%nativemode:willprobeIRQslaterPCI:Settinglatencytimer ofdevice0000:00:01.1to64 ide0:BM-DMAat0xc000-0xc007,BIOSsettings:hda:pio,hdb:pioide0:BM-DMAat0xc008-0xc00f,BIOSsettings:hdc:pio,hdd:pio ProbingIDEinterfaceide0...

hda:QEMUHARDDISK,ATADISKdrive This shows the QEMU emulated hard drive. Further on, we see: eth0:RealTekRTL8139at0xc200,00:16:3e:0b:7c:f0,IRQ11eth0:Identified 8139chiptype'RTL-8139'

The difference between PV and HVM domains doesn't end in the domU, either. With HVM domains, you'll see tap tap devices in the dom0. devices in the dom0.

Think of the tap devices as QEMU a.n.a.logues to the vifs discussed earlier-bits shoved into them come out on the virtual network devices in the domU. You can manage them just like vifs-adding them to bridges, configuring them down or up, and so on.

Paravirtualized Drivers The best solution to the problem of slow emulated HVM devices, in the context of Xen, is to use paravirtualized drivers that work on the same split-driver model as non-HVM domains-backend drivers in domain 0, with a small frontend driver in domU that communicates with the backend via event channels, using ring buffers and page flipping. (See Chapter1 Chapter1 for boring details on Xen's split driver model.) for boring details on Xen's split driver model.) XenSource includes such drivers for Windows, of course. For users of open source Xen, Novell distributes an expensive driver package that performs the same function-paravirtualized device support for Windows. GPL drivers also exist for Windows on open source Xen (we discuss those in more detail in Chapter13 Chapter13).

However, you can also build PV drivers for Linux HVM domains. These drivers are included with the Xen source tree in the unmodified_drivers unmodified_drivers directory. Unfortunately, the kernel API keeps changing, so the PV drivers might not compile against your kernel version (the drivers with Xen 3.1 refused to compile with kernel versions 2.6.20 and above). directory. Unfortunately, the kernel API keeps changing, so the PV drivers might not compile against your kernel version (the drivers with Xen 3.1 refused to compile with kernel versions 2.6.20 and above).

Compiling PV Drivers for HVM Linux Nonetheless, the best way to figure out if the drivers will work is to try it. Here's how we compiled our drivers out of a standard Xen source tree.

#cdunmodified_drivers/linux-2.6#./mkbuildtree #make-C/usr/src/linuxM=$PWDmodules This builds standard Xen devices as modules-install them into your modules tree and load them like any other driver, with insmod insmod or or modprobe modprobe. You'll wind up with four modules, one each for block and network devices, one for xenbus, and one for PCI emulation. Load xen-platform-pci first, then xenbus, then block and network.

#insmodxen-platform-pci #insmodxenbus #insmodxenblk #insmodxennet Because we were using Slackware as our domU, we then built a minimal kernel-without drivers for the emulated IDE or Realtek network card-and built an initrd with the Xen devices included.

We also needed to modify /etc/fstab /etc/fstab to refer to the Xen backend devices. to refer to the Xen backend devices.

Finally (this is starting to seem like a lot of trouble, isn't it?) we edited the domain's configuration to specify netfront and blkfront instead of the ioemu devices. We did this by changing the device lines: vif=['type=ioemu,bridge=xenbr0']

disk=['file:/opt/florizel.img,ioemu:hda,w','file:/root/slackware-12.0- install-dvd.iso,ioemu:hdc:cdrom,r']

to: vif=['bridge=xenbr0']

disk=['file:/opt/florizel.img,hda,w','file:/root/slackware-12.0-install- dvd.iso,hdc:cdrom,r']

and removing the device_model= device_model= line. line.

Modify these directions to work with your setup, of course.

And, for Our Next Trick...

As always, there are some areas of ongoing work. Both Intel and AMD have announced successor techniques for dealing with guest page tables by adding a new level to the page table structure. AMD terms the concept nested paging nested paging, while Intel calls it Extended Page Tables Extended Page Tables. IOMMU development is another exciting area of research.

HVM is nice in general, but of course all of this is a prelude to Chapter13 Chapter13, where we'll apply all of this stuff to getting a virtual Windows machine up and running in HVM mode. Stay tuned!

Chapter13.XEN AND WINDOWS

In the last chapter we described Xen's hardware virtualization support and how to use it. Now that we have Xen operating with hardware virtualization, we can run unmodified operating systems, including Windows, that 800-pound gorilla of the computing world.

Why Run Windows Under Xen?

Now, why exactly would you want to do this terrible thing? We could say, "because you can," and that's reason enough to do many things. But it might not be entirely satisfying-it sounds like a lot of work, after all.

Fortunately, reasons abound. Probably the best of these is the software-especially the server software, considering that Xen's strengths are in the server field.[79] Much of the Windows server software, like Exchange Server, has a giant installed base and would be difficult to replace. The same holds for the client software. Take, for example, Office, Outlook, Visual Studio-Microsoft continues to be a fact of life. Xen with VNC, SDL, or rdesktop allows you to run productivity software at acceptable speeds, with full, albeit unaccelerated, graphics support, all while retaining your comfortable Linux computing environment. Much of the Windows server software, like Exchange Server, has a giant installed base and would be difficult to replace. The same holds for the client software. Take, for example, Office, Outlook, Visual Studio-Microsoft continues to be a fact of life. Xen with VNC, SDL, or rdesktop allows you to run productivity software at acceptable speeds, with full, albeit unaccelerated, graphics support, all while retaining your comfortable Linux computing environment.

Xen also gives you a way out from the old worry that the next security fix or driver update will leave the machine completely unbootable-it provides an equivalent to Microsoft's "system restore points" but under your own control, rather than the OS's. By running Windows in a virtual machine with an effective storage backend, you can snapshot it, roll back updates, and protect yourself from malware-all without involving the domU.

Security furnishes another reason to run Windows under Xen. Although it's cliche to say that Windows is insecure, the fact remains that virtualization, when properly used, can be an effective layer of isolation for any server, Windows, *nix, or otherwise. The fact that Windows is a popular target makes this reasoning even more compelling.

It's also worth noting that Xen's design, which part.i.tions drivers in isolated domains, at least contributes to security. An intrusion in the Windows domain, even if it manages to exploit driver code to interact with physical hardware, is unlikely to compromise other domains on the same hardware. This doesn't mean that Xen is inherently secure, but it does suggest that it might be possible to secure it. Work remains to be done, of course-indeed, a lot of the present work in QEMU is in precisely the area of validating emulated device access to improve security. But even as things are, Xen can help to reduce the exposure of Windows machines.

Besides, it's easier to run Windows in a domU than you might suppose.

[79] At least in light of the ongoing commodification of the operating system, which really is at least part of what Xen is all about, isn't it? At least in light of the ongoing commodification of the operating system, which really is at least part of what Xen is all about, isn't it?

Windows on Xen: Prerequisites Before stepping into the brave new world of virtualized Windows, make sure that you've got a few things.

First, you're going to need a box that does HVM. (See Chapter12 Chapter12 for details on what that is, how it works, and how to tell if you have it.) The box will also need enough free memory and storage for a Windows install-Xen can help to use resources more efficiently, but it's not magic. For example, we suggest that a Xen box running Windows Server 2003 domUs have 512MB of memory and about 10GB of hard drive s.p.a.ce for each domU, plus 512MB and another 10GB for dom0. You might want to consult Microsoft's website for more specific system requirements or requirements for other Windows versions. for details on what that is, how it works, and how to tell if you have it.) The box will also need enough free memory and storage for a Windows install-Xen can help to use resources more efficiently, but it's not magic. For example, we suggest that a Xen box running Windows Server 2003 domUs have 512MB of memory and about 10GB of hard drive s.p.a.ce for each domU, plus 512MB and another 10GB for dom0. You might want to consult Microsoft's website for more specific system requirements or requirements for other Windows versions.

It might sound obvious, but you're also going to need a copy of Windows, with a license that allows virtualization. This copy of Windows should be reasonably new. We're going to focus on Windows XP Service Pack 2 and Windows Server 2003 in this chapter-Vista's just too much ha.s.sle at this point. Windows NT and 9x, while they should in theory work, seem to crash during the install step. No one (us included) seems too interested in figuring out why this happens. Life is full of mysteries.

Windows on Xen: Installation Windows is actually quite easy to get installed and running. The basic steps are exactly like any other Xen install, with some extra configuration: 1. Get Windows install media and put it somewhere that Xen can find it.

2. Make a config file.

3. Boot from the install media.

4. Install using the Windows installer.

5. Reboot from the emulated hard drive.

6. Install paravirtualized drivers.

7. Use Windows.