The Book of Xen Part 10

Both Solaris Express and OpenSolaris incorporate Xen support. Solaris Express has the Xen packages included on the DVD, while OpenSolaris requires you to download Xen as an add-on. Both of them provide a fairly polished experience. Although there are other distros based on the released Solaris code, none of them are particularly Xen-oriented, so the officially blessed distros are probably the best place to start.

RUNNING SOLARIS EXPRESSWe had some trouble deciding whether to focus on OpenSolaris or Solaris Express while we were writing this chapter. We decided to go with OpenSolaris because it seemed more popular, based on a completely unscientific poll of our friends.However, Solaris Express is still a perfectly fine OS with excellent Xen support, so we've also included some notes on setting it up.Believe it or not, Xen support should exist pretty much out of the box.[47] When you install Solaris Express on a system that supports Xen, it installs a Xen kernel and gives you the option to boot it from GRUB-just select Solaris xVM and off you go. (The included Xen version is 3.1.4 as of snv_107.) When you install Solaris Express on a system that supports Xen, it installs a Xen kernel and gives you the option to boot it from GRUB-just select Solaris xVM and off you go. (The included Xen version is 3.1.4 as of snv_107.)From there, you can install domUs normally. It's even got virt-manager virt-manager. Take a look at the next section for more details on setting up domUs. Most of these steps will apply to Solaris Express and OpenSolaris equally well.

In general, there are three possible configurations of (Open)Solaris that are of interest in our discussion of Xen.

First, we have the Solaris dom0.

Second, there's the Solaris domU on a Solaris dom0. This is a fairly straightforward setup.

Finally, you can run a Solaris domU under Linux with a minimum[48] of fuss. of fuss.

Solaris Dom0 Let's start by setting up an OpenSolaris dom0, since you'll need one for the next section. (Although we suppose this applies only if you're doing something crazy like running through all our examples in order.) Note that we're going to be using pfexec pfexec, the Solaris equivalent of sudo sudo,[49] for these examples, so it's not necessary to be root for these steps. for these examples, so it's not necessary to be root for these steps.

First, download the distribution from http://opensolaris.org/os/downloads/. Follow the directions to unpack and burn it, and boot from the CD, just like virtually any other OS install.

The OpenSolaris LiveCD will probably be a familiar experience to anyone who's installed Ubuntu. It 's really quite similar, with a standard GNOME desktop, some productivity software, and a cute Install OpenSolaris Install OpenSolaris icon on the desktop. Double-click the Install OpenSolaris icon to launch the installer, then follow its directions. icon on the desktop. Double-click the Install OpenSolaris icon to launch the installer, then follow its directions.

When the installer finishes, it'll prompt you to reboot.

Setting Up Xen If, once you reboot, you notice that you don't have Xen available, don't panic. OpenSolaris, unlike Solaris Express, doesn't include the Xen packages in the initial install. (Everything had to fit on a CD, after all.) You will have to install and set them up manually.

First, we create a ZFS boot environment. (If you're not familiar with boot environments, subst.i.tute the word snapshot snapshot. The idea is, if you break your system trying to install Xen, you can reboot into the original environment and try again.) $pfexecbeadmcreate-a-dxvmxvm $pfexecbeadmmountxvm/tmp/xvm Next, we use the OpenSolaris pkg pkg command to install the Xen packages in the new boot environment. command to install the Xen packages in the new boot environment.

$pfexecpkg-R/tmp/xvminstallxvm-gui As of OpenSolaris 2008.11, the xvm-gui package cl.u.s.ter provides all the necessary Xen packages. Previous versions may require you to install the packages individually. If you need to do that, you should be able to get away with running: #pkginstallSUNWxvmhvm #pkginstallSUNWvirtinst #pkginstallSUNWlibvirt #pkginstallSUNWurlgrabber These packages provide Xen (with HVM), virt-install virt-install, and virt-install virt-install 's dependencies. 's dependencies.

Next, we need to update GRUB to boot the Xen kernel properly.

Under OpenSolaris, menu.lst menu.lst is at is at /rpool/boot/grub/menu.lst /rpool/boot/grub/menu.lst. Edit the xvm menu item to look something like the following: t.i.tlexvm findroot(pool_rpool,0,a) bootfsrpool/ROOT/xvm kernel$/boot/$ISADIR/xen.gz module$/platform/i86xpv/kernel/$ISADIR/unix/platform/i86xpv/kernel/$ISADIR/ unix-B$ZFS-BOOTFS,console=text module$/platform/i86pc/$ISADIR/boot_archive Note that we're using extensions to GRUB that enable variables in menu.lst menu.lst, such as $ISADIR $ISADIR (for Instruction Set Architecture). Apart from that, it's a fairly normal Xen GRUB config, with the hypervisor, kernel, and ramdisk. (for Instruction Set Architecture). Apart from that, it's a fairly normal Xen GRUB config, with the hypervisor, kernel, and ramdisk.

Reboot.

Solaris SMF When you begin to configure a Solaris dom0, you'll probably notice immediately that some files aren't quite where you expect. For one thing, Solaris doesn't have an /etc/xen /etc/xen directory, nor does it have the customary scripts in directory, nor does it have the customary scripts in /etc/init.d /etc/init.d. The various support scripts in /etc/xen/scripts /etc/xen/scripts instead live in instead live in /usr/lib/xen/scripts /usr/lib/xen/scripts. You can keep domain configurations wherever you like. (We actually make an /etc/xen /etc/xen directory and put domain configurations in it.) directory and put domain configurations in it.) Instead of relying on the standard Xen config files, Solaris handles configuration and service startup via its own management framework, SMF (Service Management Facility). You can examine and change xend xend 's settings using the 's settings using the svccfg svccfg command: command: #svccfg-sxendlistprop This will output a list of properties for the xend xend service. For example, to enable migration: service. For example, to enable migration: #svccfg-sxendsetpropconfig/xend-relocation-address=""

#svcadmrefreshxend #svcadmrestartxend You may have to enable the Xen-related services manually using svcadm svcadm, particularly if you initially booted the non-Xen kernel. To look at which services are stopped, use svcs svcs: #svcs-xv If the Xen services are stopped for maintenance or disabled, you can enable them using svcadm svcadm: #svcadmenablestore #svcadmenablexend #svcadmenablevirtd #svcadmenabledomains #svcadmenableconsole From that point, you should be able to use Solaris as a perfectly normal dom0 OS. It's even got libvirt. Have fun.

Creating a Solaris DomU You didn't really think it would be that easy, did you? There are a couple of small caveats to note-things that make Xen under Solaris a slightly different animal than from under Linux. We'll start by creating a Solaris domU on a Solaris dom0, then extend our discussion to a Solaris domU on a Linux dom0.

ZFS Backing Devices First, we suggest handling virtual block devices a bit differently under Solaris. Although you can create domU filesystems as plain loopback-mounted files, ZFS is probably a better option. It's been praised far and wide, even winning some grudging accolades from Linus Torvalds. It is, in fact, ideal for this sort of thing, and the generally accepted way to manage disks under Solaris-even more so now that OpenSolaris uses a ZFS root filesystem.

ZFS is pretty simple, at least to get started with. Users of LVM should find that creating a pool and filesystem are familiar tasks, even though the commands are slightly different. Here we'll make a pool, create a ZFS filesystem within the pool, and set the size of the filesystem: #zpoolcreateguestsc0d0 #zfscreateguests/escalus #zfssetquota=4gguests/escalus Now we can define a domain that uses the phy: phy: device device /dev/zvol/dsk/guests/escalus /dev/zvol/dsk/guests/escalus for its backing store, as shown in the config file. for its backing store, as shown in the config file.

We'll leave further subtleties of ZFS administration to Sun's doc.u.mentation.

Installing a DomU via PyGRUB The last thing to do before creating the domU is to write an appropriate config file. Here's ours: #cat/etc/xen/escalus name="escalus"

memory=512 disk=[ 'file:/opt/xen/install-iso/os200805.iso,6:cdrom,r', 'phy:/dev/zvol/dsk/guests/escalus,0,w'

vif=['']

bootloader='pygrub'

kernel='/platform/i86xpv/kernel/unix'

ramdisk='boot/x86.microroot'

extra=/platform/i86xpv/kernel/unix-Bconsole=ttya,livemode=text on_shutdown='destroy'

on_reboot='destroy'

on_crash='destroy'

Note that the disk specifier works differently than with Linux domUs. Rather than using symbolic device names, as under Linux: disk=['file:/export/home/xen/solaris.img,sda1,w']

root="/dev/sda1"

we instead specify the disk number: disk=['phy:/dev/zvol/dsk/guests/ecalus,0,w']

root="/dev/dsk/c0d0s0"

Here we're installing Solaris from an ISO image (os200805.iso) using PyGRUB to pull the correct kernel and initrd off the CD image, boot that, and proceed with a normal install.

NoteOne thing to watch out for is that domU networking will only work if you're using a GLD3-based network driver. The drivers that ship with Solaris are all fine in this regard-however, you may have trouble with third-party drivers.

Once the install's done, we shut the machine down and remove the disk entry for the CD.

At this point your Solaris domU should be ready to go. Setting up a Linux domU is equally straightforward, since standard Linux domU images and kernels should work unmodified under Solaris.

Next, we'll look at setting up a Solaris domU under Linux.

Creating a Solaris DomU Under Linux For the most part, a domU is independent of the dom0 OS, and thus the install under Linux uses much the same installation procedure as under Solaris. There are only a few pitfalls for the unwary.

First, you might have a bit more work to do to ensure that the domain can find an appropriate kernel. The Solaris image will complain bitterly, and in fact will not boot, with a Linux kernel.

If you're using PyGRUB on a Xen 3.1 or later system, you shouldn't need to do anything special. PyGRUB itself will load the appropriate files from OpenSolaris installation media without further intervention, just as in the previous example.

If you're not using PyGRUB, or if you're using the stock RHEL5.1 hypervisor, you'll need to extract the kernel and miniroot (initrd, for Linux people) from the OpenSolaris install package and place them somewhere that Xen can load them.

#mount-oloop,roosol200811.iso #cp/mnt/cdrom/boot/platform/i86pv/kernel/unix/xen/kernels/solaris/ #cp/mnt/cdrom/x86.miniroot/xen/kernels/solaris/ #umount/mnt/cdrom Just as under Solaris, begin by writing a config file. We'll set up this config file to load the installer from the CD, and later alter it to boot our newly installed domU. Note that we're grabbing the kernel from the ISO, using the kernel and ramdisk options to specify the files we need.

bootloader='/usr/bin/pygrub'

kernel="/platform/i86xpv/kernel/amd64/unix"

ramdisk="/boot/x86.microroot"

extra="/platform/i86xpv/kernel/amd64/unix--nowin-Binstall_media=cdrom"

cpu_weight=1024 memory=1024 name="rosaline"

vif=['vifname=rosaline,ip=192.0.2.136,bridge=xenbr0,mac=00:16:3e:59:A7:88']

disk=[ 'file:/opt/distros/osol-0811.iso,xvdf:cdrom,r', 'phy:/dev/verona/rosaline,xvda,w'

Make sure to create your backing store (/dev/verona/rosaline in this case). in this case).

Now create the domain. Next step, installation.

Although OpenSolaris has a perfectly functional console when running as a domU, it unfortunately does not include a text mode installer. It does, however, include a VNC server and SSH server, either of which can be used to get a remote graphical display. Here's how to set up VNC.

Log in at the domU console with username jack jack and pa.s.sword and pa.s.sword jack jack.

Once you're in locally, set up your network. (If you're using DHCP, it'll probably already be set up for you, but it doesn't hurt to make sure.) #pfexecifconfigxnf0 xnf0:flags=201000843mtu1500index2 inet192.0.2.128netmaskffffff00broadcast192.0.2.255 etheraa:0:0:59:a7:88 You can see that our network is in fine shape, with the address 192.168.2.128. If it's not set up already, a.s.sign an address manually: pfexecifconfigxnf0192.0.2.128/24 The VNC server should already be running. To enable remote access to it, run the vncpa.s.swd vncpa.s.swd command: command: pfexecvncpa.s.swd/etc/X11/.vncpa.s.swd vncpa.s.swd will ask you to make up a pa.s.sword and enter it twice. Use this pa.s.sword to connect to the VNC server using your favorite VNC client. You should be greeted with an OpenSolaris desktop. will ask you to make up a pa.s.sword and enter it twice. Use this pa.s.sword to connect to the VNC server using your favorite VNC client. You should be greeted with an OpenSolaris desktop.

Finally, click the Install OpenSolaris icon on the desktop and proceed with the graphical install.

OpenSolaris DomU Postinstall Configuration Once the installer has done its work, you'll be ready to shut down the domain and move to the next step: setting up the dom0 to load a kernel from a ZFS filesystem.

The problem is that in Xen 3.3, PyGRUB's version of libfsimage libfsimage isn't able to handle recent versions of ZFS directly. Our solution was to download the Xen-unstable source tree (as of this writing, Xen 3.4-rc) from isn't able to handle recent versions of ZFS directly. Our solution was to download the Xen-unstable source tree (as of this writing, Xen 3.4-rc) from http://xenbits.xen.org/ and build PyGRUB from that. (Alternatively, you can mount the install media, extract the kernel and microroot, specify these manually in the config file, and pa.s.s the correct "extra" line to the kernel-that works just as well.) and build PyGRUB from that. (Alternatively, you can mount the install media, extract the kernel and microroot, specify these manually in the config file, and pa.s.s the correct "extra" line to the kernel-that works just as well.) #hgclonehttp://xenbits.xen.org/xen-unstable.hg #cdxen-unstable #maketools #cdxen-unstable.hg/tools/pygrub;makeinstall #cdxen-unstable.hg/tools/libfsimage;makeinstall Now we update the domain config file. Since we went to all the trouble of updating PyGRUB, we'll use it directly here: bootloader='pygrub'

cpu_weight=1024 memory=1024 name="rosaline"

vif=['vifname=rosaline,ip=192.0.2.136,bridge=xenbr0,mac=00:16:3e:59:A7:88']

disk=[ #'file:/opt/distros/osol-0811.iso,xvdf:cdrom,r', 'phy:/dev/verona/rosaline,xvda,w'

]NotePV-GRUB, at this time, isn't able to load an OpenSolaris kernel properly. Use PyGRUB instead.

Start your new domain as usual with xm xm: #xmcreaterosaline

[45] We tend to use the terms We tend to use the terms Zone Zone and and Container Container interchangeably. Technically, a Solaris Container implements system resource controls on top of Zones. interchangeably. Technically, a Solaris Container implements system resource controls on top of Zones.

[46] The CDDL is a free software license that's GPL-incompatible but generally inoffensive. The CDDL is a free software license that's GPL-incompatible but generally inoffensive.

[47] That's another reason we gloss over Solaris Express: Focusing on it would not, in the words of Douglas Adams, "make for nice fat books such as the American market thrives on." That's another reason we gloss over Solaris Express: Focusing on it would not, in the words of Douglas Adams, "make for nice fat books such as the American market thrives on."

[48] The temptation exists to write "elegant minimum," but it's simply not so. The temptation exists to write "elegant minimum," but it's simply not so.

[49] Anyone planning to take offense to the comparison of Anyone planning to take offense to the comparison of pfexec pfexec and and sudo sudo: Please a.s.sume that we have been utterly convinced by your rhetoric and carry on with your day-to-day life.

NetBSD NetBSD is a popular choice for a dom0 OS because of its small and versatile design, which is a good match for the dedicated virtualization server dedicated virtualization server model that Xen encourages. In our experience, a dom0 running NetBSD will use less memory and be at least as stable as one running Linux. model that Xen encourages. In our experience, a dom0 running NetBSD will use less memory and be at least as stable as one running Linux.

However, Linux people often make the mistake of a.s.suming that NetBSD is exactly like Linux. It's not-it's kind of close, but NetBSD is the product of an evolution as long as Linux's, and it requires some practice to work with. In this section, we're going to a.s.sume that you're familiar with NetBSD's idiosyncrasies; we're only going to cover the Xen-related differences.

NetBSD's Historical Xen Support NetBSD has supported Xen for a very long time-since NetBSD version 3.0, which incorporated support for Xen2 as dom0 and as a domU. This Xen2 support is quite stable. However, it has the obvious drawback of being Xen2, which lacks the Xen3 features like live migration and HVM. It's also 32 bit only, and doesn't support PAE (Physical Address Extension). (We've used this version quite a bit. The first Xen setup we used for hosting at prgmr.com was a dual Xeon running NetBSD 3.1 and Xen2, supporting Linux and NetBSD domUs.) NetBSD 3.1 introduced support for Xen 3.0.x-but only as a domU.

NetBSD 4.0 added Xen 3.1 support as both a domU and a dom0, and it also introduced support for HVM. The only remaining problem with NetBSD 4.0 was that it, like its predecessors, it did not support PAE or x86_64, which means that it was unable to use more than 4GB of memory. It also could not run as a domU on a 64-bit or PAE system, such as is used by Amazon's EC2. That last bit was the real killer-it meant NetBSD 4 required a non-PAE 32-bit hypervisor, which in turn limited you to 4GB of address s.p.a.ce, which translates about 3.5GB of physical memory. (This limitation is so significant that Xen.org doesn't even distribute a non-PAE binary package anymore.) Finally, the new and shiny NetBSD 5 adds PAE support for NetBSD domUs, x86-64 support for both dom0 and domUs, and support for 32-bit domUs on 64-bit dom0s (32-on-64 in Xen parlance). Work is still being done to add features to bring NetBSD Xen support into feature parity with Linux's Xen support, but NetBSD is already a perfectly viable platform. in Xen parlance). Work is still being done to add features to bring NetBSD Xen support into feature parity with Linux's Xen support, but NetBSD is already a perfectly viable platform.

Installing NetBSD as a Dom0 The basic steps to get started using NetBSD with Xen are pretty much the same as for any other OS: Download it, install it, and make it work. Again, we're a.s.suming that you're familiar with the basic NetBSD install procedure, so we're just going to outline these directions briefly.

Begin by downloading NetBSD and installing it as usual. (We opted to download and burn the ISO at http://mirror.planetunix.net/pub/NetBSD/iso/5.0/amd64cd-5.0.iso.) Configure the system according to your preference.

Noteftp:// and and http:// http:// are interchangeable on all of the are interchangeable on all of the ftp.netbsd.org URLs URLs. http:// gets through firewalls better, and gets through firewalls better, and ftp:// ftp:// is slightly faster. Pick one. Also, you often get significantly better speeds using a mirror rather than the netbsd.org site. If your FTP install fails partway through, the first thing to do is to try another mirror is slightly faster. Pick one. Also, you often get significantly better speeds using a mirror rather than the netbsd.org site. If your FTP install fails partway through, the first thing to do is to try another mirror.

However you install NetBSD, go through the installer and reboot into your new system. Next, install the Xen kernel and supporting tools using the NetBSD ports system, pkgsrc pkgsrc. Get pkgsrc pkgsrc at at http://ftp.netbsd.org/pub/NetBSD/packages/pkgsrc.tar.gz. Untar pkgsrc.tar.gz pkgsrc.tar.gz, then install Xen: #cdpkgsrc/sysutils/xenkernel3;makeinstall #cdpkgsrc/sysutils/xentools3;makeinstall After installing the Xen tools, NetBSD will remind you to create the Xen device nodes: #cd/dev;shMAKEDEVxen Now that Xen is installed, our next task is to install GRUB in place of the standard NetBSD bootloader so that we can perform the multistage boot that Xen requires: #cdpkgsrc/sysutils/grub;makeinstall Our next step is to download and install NetBSD Xen kernels-we're already running off standard NetBSD kernels, and we've got the hypervisor installed, but we still need kernels for the dom0 and domUs. Download netbsd-XEN3_DOM0.gz netbsd-XEN3_DOM0.gz, netbsd-XEN3_DOMU.gz netbsd-XEN3_DOMU.gz, and netbsd-INSTALL_XEN3_DOMU.gz netbsd-INSTALL_XEN3_DOMU.gz from your favorite NetBSD mirror. (We used from your favorite NetBSD mirror. (We used http://mirror.planetunix.net/pub/NetBSD/NetBSD-5.0/.) Now that we have suitable Xen kernels to go with the hypervisor and supporting tools that we installed in the previous step, we can set up GRUB in the usual way: #grub-install--no-floppysd0a Edit /grub/menu.lst /grub/menu.lst so that it boots the Xen kernel and loads NetBSD as a module. Here's a complete file, with comments (adapted from a NetBSD example at so that it boots the Xen kernel and loads NetBSD as a module. Here's a complete file, with comments (adapted from a NetBSD example at http://www.netbsd.org/ports/xen/howto.html): #Bootthefirstentrybydefault default=1

#after10s,bootthedefaultentryiftheuserdidn'thitkeyboard timeout=10

#Configureserialporttouseasconsole.Ignorethisbitifyou're #notusingtheserialport.

serial--unit=0--speed=115200--word=8--parity=no--stop=1

#Lettheuserselectwhichconsoletouse(serialorVGA).Default #toserialafter10s terminal--timeout=10consoleserial #AnentryforNetBSD/xen,using/xen/kernels/xen.gzasthedomain0 #kernel,withserialconsole.Domain0willhave64MBRAMallocated.

#a.s.sumeNetBSDisinstalledinthefirstMBRpart.i.tion.

t.i.tleXen3.3/NetBSD(sd0a,serial)root(hd0,0)kernel (hd0,a)/xen/kernels/xen.gzdom0_mem=65536com1=115200,8n1module (hd0,a)/xen/kernels/XEN3_DOM0root=sd0aroconsole=ttyS0

#Sameasabove,butusingVGAconsole #Weca.n.u.seconsole=tty0(Linuxsyntax)orconsole=pc(NetBSDsyntax) t.i.tleXen3.3/NetBSD(sd0a,vga) root(hd0,0) kernel(hd0,a)/xen/kernels/xenkernel3-3.1.0nb2dom0_mem=65536noreboot module(hd0,a)/xen/kernels/XEN3_DOM0root=sd0aroconsole=pc

#LoadaregularNetBSD/i386kernel.Canbeusefulifyouendupwitha #nonworking/xen.gz t.i.tleNetBSD5 root(hd0,a) kernel(hd0,a)/netbsd-GENERIC The important bits are the kernel name, XEN3_DOM0 XEN3_DOM0, and the root device, which we specify using NetBSD syntax.

NoteWe've also set up this config file to use the serial console. No matter which operating system you use, we strongly recommend using a serial console with Xen, even if you prefer to use a KVM or other method of remote management normally. See Chapter14 Chapter14 for more discussion of the many and varied uses of the serial console for more discussion of the many and varied uses of the serial console.

Copy over the basic Xen config files to the directory where the Xen tools will expect to find them: #cp/usr/pkg/share/examples/xen/*/usr/pkg/etc/xen/ Now that we have all the parts of a NetBSD dom0, we need to start xenbackendd xenbackendd and and xend xend (in that order, or it won't work). (in that order, or it won't work).

#cp/usr/pkg/share/examples/rc.d/xen*/etc/rc.d/