The Book of Xen Part 6

gateway="38.99.2.1"

ip="38.99.2.222"

Note that we're just adding extra Linux kernel configuration to the domain config-values not used by Xen will be pa.s.sed to the kernel command line. You can also explicitly put this configuration in the "extra" parameter. If you want to set the IP address via DHCP, you can replace the last three lines above with: dhcp="dhcp"

You can then use DHCP to specify the NFS server and NFS root as well in the usual manner. Boot the domain and you're done. Because the storage is accessible to any machine on the network, Xen's live migration should work as well.

Closing Suggestions This might seem like a bewildering, or even excessive, variety of storage options, but all of these have their places-be it in a hosting environment, or on the desktop, or in a storage pool for utility computing. The recommendations we've made in this chapter are a start, but in the end the best advice we can offer is to try all of these and see what works best. Find the right trade-off between ease of administration and scalability.

Finally, you can combine and extend many of these options. For example, the Xen LiveCD uses flat images with LVM snapshots. Depending on your application, the best solution might be simple filesystem images or a combination of software RAID and LVM. Keep experimenting and see what fits best. These are all examples of the flexibility of Xen's standards-based architecture, which relies on user-extensible scripts to define available storage using easily understood semantics. In Chapter5 Chapter5, we'll look at how these same principles apply to Xen's network setup.

Chapter5.NETWORKING

So you've got your nice, shiny Xen box all set up and running, and now you want it to talk to the outside world. This seems eminently reasonable. After all, Xen's primary use is server consolidation-and a non-networked server is a contradiction in terms. Well, you're in luck. Xen has a robust and well-tested network foundation, which is versatile and easy to set up.

Like much of the rest of Xen, the networking infrastructure reuses as many standard tools as possible. In this case, Xen uses the standard bridge utilities and the ip ip command command[33] all glued together with some clever bash and Python scripts to handle traffic between the dom0 and domU network interfaces. all glued together with some clever bash and Python scripts to handle traffic between the dom0 and domU network interfaces.

As an administrator, your primary interaction with Xen virtual net devices is through the various config files. As you might expect, global parameters and initial setup are mostly handled by directives in xend-config.sxp xend-config.sxp. Domain-specific configuration is done within the domain's config file.

More advanced configuration can be done by modifying Xen's network scripts, stored in /etc/xen/scripts /etc/xen/scripts.

If you are the sort that avoids a reboot at all costs, you can often directly manipulate Xen's network infrastructure while the VM is running through brctl, iptables brctl, iptables, and the xm xm command, but these changes don't always successfully propagate to the domU. We will focus on the "manipulate the config file and reboot the domU" method here because it works in all situations. command, but these changes don't always successfully propagate to the domU. We will focus on the "manipulate the config file and reboot the domU" method here because it works in all situations.

Xen's Network Setup Process Xen runs its network scripts at two points: at xend xend startup and at domain startup. (There are corresponding scripts for domain shutdown and startup and at domain startup. (There are corresponding scripts for domain shutdown and xend xend stop.) stop.) The role of the script run at xend xend startup (usually either startup (usually either network-bridge network-bridge or or network-route network-route) is to switch from standard, non-Xen networking to Xen-based networking. In the case of network-bridge network-bridge, for example, this script creates a bridge and a.s.signs the physical Ethernet device to it. Then it initializes dom0's networking, creates a virtual interface, adds it to the bridge, and copies the configuration over to the bridged network device.

When Xen starts a domain, it executes the appropriate vif-* vif-* script-for example, script-for example, vif-bridge vif-bridge in the case of in the case of network-bridge network-bridge. The following shows the various scripts that would be run with the default setup.

(xendstartup) -network-bridge -xen-script-common.sh -xen-network-common.sh

(xmcreate) -vif-bridge -vif-common.sh -xen-hotplug-common.sh -logging.sh -xen-script-common.sh -locking.sh -xen-network-common.sh Most of these scripts aren't terribly important to our purposes at the moment. For example, logging.sh logging.sh is just sourced to provide a is just sourced to provide a log() log() function. The main points we're interested in are the ones marked with function. The main points we're interested in are the ones marked with . These are good scripts to edit or good places to introduce a wrapper script. . These are good scripts to edit or good places to introduce a wrapper script.

These scripts are bash sh.e.l.l scripts (rather than Python, as one might expect from the fact that many of the other Xen configuration files are in Python). They all live in the /etc/xen/scripts /etc/xen/scripts directory by default, alongside scripts to support the other cla.s.ses of Xen virtual devices. directory by default, alongside scripts to support the other cla.s.ses of Xen virtual devices.

[33] The IP command The IP command /sbin/ip /sbin/ip is the modern (unfortunately named) replacement for is the modern (unfortunately named) replacement for ifconfig ifconfig. Get it as part of the iproute2 tool set, which offers similar functionality to net-tools but with added features.

Defining Virtual Interfaces All of the Xen networking options work by creating virtual interfaces in dom0 to serve as targets for bridging, iptables iptables rules, and so on. Each virtual interface is defined by a section in the rules, and so on. Each virtual interface is defined by a section in the vif= vif= line of the config file, delimited by a pair of single quotes. line of the config file, delimited by a pair of single quotes.

Xen 3 supports up to eight virtual interfaces per domain (three prior to 3.1.) For example, vif=['','','']

defines three virtual interfaces and tells Xen to configure them with sensible defaults. Note that they're comma-separated and and delimited, in accord with Python array syntax. delimited, in accord with Python array syntax.

You can examine these virtual interfaces from the dom0 by typing #ifconfig-a This will output information for a bunch of devices of the form vifx.y where x x and and y y are numbers. (You might see are numbers. (You might see tap tap devices as well if you're using HVM. Just treat them like vifs for now.) devices as well if you're using HVM. Just treat them like vifs for now.) NoteUnder recent Red Hatderived distros (including CentOS 5.1 and Fedora 8), libvirt will create another bridge called virbr0 virbr0. This bridge is set up for NAT by default and should be considered part of libvirt, rather than Xen. It functions much like Xen's This bridge is set up for NAT by default and should be considered part of libvirt, rather than Xen. It functions much like Xen's network-nat network-nat implementation: dom0 runs a dnsmasq server on implementation: dom0 runs a dnsmasq server on virbr0 virbr0 and NAT's packets for domUs. It's a sensible default for a desktop, but probably ill-suited to a server. Specify and NAT's packets for domUs. It's a sensible default for a desktop, but probably ill-suited to a server. Specify bridge=xenbr0 bridge=xenbr0 to use Xen's bridge, rather than libvirt's. For the curious to use Xen's bridge, rather than libvirt's. For the curious, virbr virbr is configured by is configured by /etc/libvirt/qemu/networks/default.xml. /etc/libvirt/qemu/networks/default.xml. You can disable the You can disable the virbr virbr devices by removing the symlink in devices by removing the symlink in /etc/libvirt/qemu/networks/autostart. /etc/libvirt/qemu/networks/autostart.

Xen's default behavior is to give each virtual interface a name based on the domain number, where x x is the domain number and is the domain number and y y is the sequential number of the interface in the domU. For example, is the sequential number of the interface in the domU. For example, vif2.0 vif2.0 is is eth0 eth0 in domain 2; in domain 2; vif2.1 vif2.1 is is eth1 eth1 in domain 2. In addition, each physical network card in the machine will show up as a in domain 2. In addition, each physical network card in the machine will show up as a peth peth device (for device (for physical Ethernet physical Ethernet). xend xend creates these virtual interfaces on behalf of domains on domain startup. Thus, you'll see a varying number of them, depending on how many domains are running and how many network devices they have. creates these virtual interfaces on behalf of domains on domain startup. Thus, you'll see a varying number of them, depending on how many domains are running and how many network devices they have.

As we've stated before, the dom0 is a domain just like the rest, except for its ability to perform control-plane functions and its access to PCI devices. Therefore, vif0.0 vif0.0 is domain 0's virtual is domain 0's virtual eth0 eth0 interface, and interface, and vif0.1 vif0.1 is is eth1 eth1 in the dom0. (Xen conveniently creates an alias from in the dom0. (Xen conveniently creates an alias from eth ethX to to vif0. vif0.x.) Noteveth devices also show up in the dom0. Don't worry about them. They're just used as scratch paper while devices also show up in the dom0. Don't worry about them. They're just used as scratch paper while xend xend copies information around copies information around.

Ordinarily you can ignore the vifs while interacting with dom0. However, there are a few things to note: Because eth0 eth0 in dom0 is actually an alias for a virtual interface, it only sees traffic directed at dom0-domU traffic never reaches the alias. If you want to see the traffic going to the domUs with in dom0 is actually an alias for a virtual interface, it only sees traffic directed at dom0-domU traffic never reaches the alias. If you want to see the traffic going to the domUs with tcpdump tcpdump or similar in dom0, you must look at the physical Ethernet card-that is, the appropriate or similar in dom0, you must look at the physical Ethernet card-that is, the appropriate peth peth interface. interface.

Also, the peth peth might have a strange MAC, usually FF:FF:FF:FF:FF:FF. Because no packets actually originate or terminate at the might have a strange MAC, usually FF:FF:FF:FF:FF:FF. Because no packets actually originate or terminate at the peth peth device, it doesn't need an actual MAC address. device, it doesn't need an actual MAC address.

Finally, the virtual devices are provided by the netloop driver, and it defaults to only allowing eight virtual devices at a time. This value can be raised by increasing the value of the nloopbacks nloopbacks parameter: parameter: #modprobenetloopnloopbacks=128 If it's built into the kernel, you can boot the dom0 kernel with a higher value for loopback.nloopbacks loopback.nloopbacks. Simply append netloop.nloopbacks=128 netloop.nloopbacks=128 (or a similarly large value) to the (or a similarly large value) to the module module line that loads the Linux kernel in line that loads the Linux kernel in /boot/grub/menu.lst /boot/grub/menu.lst.

Naming Virtual Interfaces Accounting for bandwidth usage with this elaborate scheme of virtual interfaces can be tricky. Because the interface names are based on the domain number, and the domain number changes each time the domain reboots, it's impractical to simply monitor bandwidth using the dom0's internal counters-you'll wind up with stuff like this: vif58.0RXbytes:12075895(11.5MiB)TXbytes:14584409(13.9MiB)RE-INDEXING NAMESIf you decide to use vifnames, you should probably note that the mib mib number of the interface might change on every reboot of the domU, though the symbolic name will remain the same. number of the interface might change on every reboot of the domU, though the symbolic name will remain the same.Some SNMP monitoring software expects mib numbers to remain the same. When they change, the software interprets this as a new interface of the same name, which makes collecting c.u.mulative statistics ... difficult. You can often fix the madness by configuring your SNMP software to aggressively reindex names to numbers, rather than relying on its cache. It's just something to watch out for-not unfixable, but annoying. Check your specific monitoring software for details.

One way around this is to name the virtual interfaces so that the name becomes independent of the domain number. You can name virtual interfaces from within the domU config file by specifying the vifname vifname parameter as part of the parameter as part of the vif= vif= configuration directive. For example, configuration directive. For example, vif=['vifname=http']

would cause the vif vif in the dom0 to become in the dom0 to become http http rather than rather than vif vifx.y. (Within the domain, of course, it just shows up as eth0 eth0.) If the domain has multiple interfaces, specify them as follows: vif=['vifname=http1','vifname=http2']

As of this writing, you'll then need to stop and start the domain to force it to reread the config file; a simple reboot won't work.

With the vifname vifname specified, we can stop and start the domain. Here we shut down a domain: specified, we can stop and start the domain. Here we shut down a domain: #xmlisthttp.xen NameIDMem(MiB)VCPUsStateTime(s) http.xen6961-b----11358.0 #xmshutdownhttp.xen xenbr0:port8(vif6.0)enteringdisabledstate devicevif6.0leftpromiscuousmode xenbr0:port8(vif6.0)enteringdisabledstate Note that the device is simply vif6.0 vif6.0. Now start it up.

#xmcreatehttp.xen Usingconfigfile"http.xen".

devicehttpenteredpromiscuousmode xenbr0:port8(http)enteringlearningstate xenbr0:topologychangedetected,propagating xenbr0:port8(http)enteringforwardingstate Starteddomainhttp.xen.prgmr.com

#cathttp.xen.prgmr.com grepvifname vif=['vifname=http,bridge=xenbr0']TRUNCATED VIFNAMESLinux silently truncates the vifname vifname to 15 characters-it'll ignore any input beyond that without raising an error. Thus, for a to 15 characters-it'll ignore any input beyond that without raising an error. Thus, for a vif named wiki.xen.prgmr.com vif named wiki.xen.prgmr.com,#ifconfigwiki.xen.prgmr.first 15 characters, including the dot) or#ifconfigwiki.xen.prgmr.comor#ifconfigwiki.xen.prgmr.foo.bar.bazwill work, but#ifconfigwiki.xenwill fail with an ENODEV ENODEV (a "no such device" or "device not found" error). (a "no such device" or "device not found" error).Note that 15-character vifnames cause problems with some versions of iptables. Keep your vifnames to 8 characters to be safe.

Autoconfiguration You can just leave the vif= vif= line blank and let line blank and let xm xm generate the entire configuration automatically, like so: generate the entire configuration automatically, like so: vif=['']

This scenario has the advantage of allowing the domU administrator to configure the network with complete freedom from within the domU, using whatever tools are most convenient. This is the most accurate way of simulating a physical machine. Xen can generate IP and MAC addresses itself, if need be, and configure them to work without administrator intervention.

However, we don't recommend this because it leaves configuration of the network interface up to the users. At a minimum, you should specify the IP address so that Xen can set up the antispoofing rules to prevent an attacker with a Xen instance faking the source or destination address in his IP header, and specify a MAC address to avoid the possiblity of conflict. An IP or MAC address conflict can take down your network! Avoid it if at all possible.

MYSTERIOUS INCREASING INTERFACE NUMBERSOne other problem that you can avoid by specifying a MAC address is the domU becoming confused when the address changes on each boot-on some systems it'll believe that each new address is a new card, so that it uses eth1 on the second boot, eth2 on the third, and so on. Upon seeing the new MAC address, udev believes it's found a new card.The easy solution is to specify a MAC address in the domain config file, and clear out udev's cache in /etc/udev/rules.d/z25_persistent-net.rules /etc/udev/rules.d/z25_persistent-net.rules. This should cause the device to come up as eth0 on the next boot and stay eth0, since it's not getting a random address anymore.

Specifying a MAC Address Ordinarily, in the absence of virtualization, hardware manufacturers do a good job of ensuring that each Ethernet device has a unique MAC address. With Xen, however, it becomes the administrator's responsibility to ensure that each virtual interface has a unique hardware address.

First, think back for a moment to the 7-layer OSI network model. Xen interacts with this model at layers 2 and 3; that is, it provides a virtual layer 2 switch with network-bridge network-bridge and functions as an IP router with and functions as an IP router with network-route network-route. Both of these layers require the Xen domain to have a unique address. The antispoof rules and ip ip directive can take care of that for layer 3; however, you are also likely going to want to specify a MAC address. directive can take care of that for layer 3; however, you are also likely going to want to specify a MAC address.

Xen can simply make one up, but this results in a relatively high probability of a collision. When Xen picks a MAC for you, it starts with the 00:16:3e prefix a.s.signed to Xen by the IEEE registration authority, and it picks the remaining three bytes at random; this means you have 3 bytes of entropy. At 1,000 hosts on one network (most of a /22) this gives you something like a 3 percent chance of a collision. (You can calculate this using the birthday paradox-a good use for your obscure math trivia.) Considering the huge yuckiness of a MAC address conflict, we suggest always manually specifying the MAC address. Do this by changing your vif= vif= line to include a line to include a mac= mac= section, like so: section, like so: vif=['ip="10.0.0.1",mac="ae:00:01:02:03:04"']

Here at prgmr.com, we pick a 2-byte prefix and append the IP address in hex because the IP address is already unique. There are some important rules when doing this, though: First, the most significant bit should be zero; second, the address should be part of the "locally a.s.signed" block to avoid possible conflicts with real Ethernet hardware. Fortunately, these rules can be distilled into a basic formula: Make the second hex digit of the first octet (e in the above example) one of 2, 6, A, or E. (Of course, you can avoid having to worry about this by using the Xen prefix mentioned above.) in the above example) one of 2, 6, A, or E. (Of course, you can avoid having to worry about this by using the Xen prefix mentioned above.)

Manipulating vifs with xm Although we ordinarily modify networking settings indirectly through the config files, it's also possible to make changes directly using xm xm.

The relevant commands are network-attach network-attach and and network-detach network-detach. Using network-attach network-attach, you can create a new virtual NIC as if it had been specified in the config file, with options specified on the command line. For example, xmnetwork-attachwiki.xen.prgmr.comscript=network-bridgemac=00:16:3e:02:ac:7dbridge=xenbr0 If you don't specify the various parameters, Xen will supply default values, just as if they'd been unspecified in the vif= vif= line of the config file. line of the config file.

Similarly, network-detach network-detach will detach a will detach a vif vif from the domU and destroy it. from the domU and destroy it.

xmnetwork-detachwiki.xen.prgmr.com0 The preceding command will remove eth0 eth0 from the domain from the domain wiki.xen.prgmr.com.

Securing Xen's Virtual Network From a security standpoint, there are two aspects of Xen networking that bear mentioning. First, we want to make sure that the users can't pretend to be someone they're not. This is addressed by specifying an IP address and enabling the antispoofing rules. Second, we want to protect the dom0 while letting traffic through to the domUs. This is easily handled by appropriate iptables iptables rules. rules.

Specifying an IP Address The antispoofing rules use iptables iptables to ensure that the Xen box will drop packets that don't match the expected address for the to ensure that the Xen box will drop packets that don't match the expected address for the vif vif that they're coming from, thus protecting your network from a rogue domU. The network scripts set this up using that they're coming from, thus protecting your network from a rogue domU. The network scripts set this up using iptables iptables to add a rule to the FORWARD chain allowing packets that match that IP address to pa.s.s to the domU's network device. (The function that does this is in to add a rule to the FORWARD chain allowing packets that match that IP address to pa.s.s to the domU's network device. (The function that does this is in vif-common.sh vif-common.sh, for the curious.) For this to work, your FORWARD chain should have a policy of DROP-network-bridge should handle that automatically. should handle that automatically.

We use antispoofing with network-bridge. network-route network-bridge. network-route adds similar rules. It's known to not work with adds similar rules. It's known to not work with network-nat network-nat.

Add the following to /etc/xen/xend-config.sxp /etc/xen/xend-config.sxp: (network-script'network-bridgeantispoof=yes') and set the following in the domain config file: vif=['ip=10.0.0.1',bridge=xenbr0]

(Use an appropriate IP and bridge for your site, obviously.) You can also specify a range of IP addresses in CIDR format (CIDR stands for stands for Cla.s.sless Inter-Domain Routing Cla.s.sless Inter-Domain Routing); that is, with a slash and the number of bits set in the netmask, in decimal. For example, to allow 10. anything, the previous line could be rewritten as vif=['ip=10.0.0.1/8']

This doesn't keep the domU administrator from changing the domU's IP address, but it does block any packets from that changed IP address.

Firewalling the Dom0 With Xen's networking, the INPUT and OUTPUT chains don't affect packets aimed at the domUs. Thus, standard firewalls on the INPUT chain, like Red Hat's, won't affect domU packets. (domUs themselves, of course, can firewall their own virtual network devices as needed.) NoteMany systems, by default, don't send bridge traffic through the FORWARD chain as you would expect-RHEL/CentOS 5.1 is an example of this problem. This causes the antispoofing rules to not work. A simple echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables echo 1 > /proc/sys/net/bridge/bridge-nf-call-iptables solves the problem. Add that line to solves the problem. Add that line to /etc/xen/scripts/network-bridge /etc/xen/scripts/network-bridge to have it run automatically when Xen sets up its networking to have it run automatically when Xen sets up its networking.

The only problem that we've seen with sending domU packets through the FORWARD chain is that, by default, the dom0 includes them in its connection tracking. On heavily loaded machines, the connection table will fill up, and the machine will drop packets. Our solution is to edit the frob_iptable() frob_iptable() function in function in vif-bridge vif-bridge to include a rule like the following: to include a rule like the following: iptables-traw"$c"PREROUTING-mphysdev--physdev-in"$vif""$@"-jNOTRACK This lets antispoof work and keeps the domU traffic from interfering with the dom0, while allowing the domUs to have unhindered access to their packets.

Networking with network-route network-route is the original option chosen by the Xen team (and few have used it since). It works by creating an internal IP router, which forwards traffic to and from the guest domains. Note that it doesn't do address translation-for that you'll want is the original option chosen by the Xen team (and few have used it since). It works by creating an internal IP router, which forwards traffic to and from the guest domains. Note that it doesn't do address translation-for that you'll want network-nat network-nat, or virbr virbr. It has largely been superseded by network-bridge network-bridge, which allows considerably more flexibility.

network-route does have its place, however. For one thing, it is transparent to the upstream network hardware, unlike does have its place, however. For one thing, it is transparent to the upstream network hardware, unlike network-bridge network-bridge. For another thing, it's much simpler. network-route network-route simply enables IP forwarding and then creates simply enables IP forwarding and then creates iptables iptables rules in the dom0 to forward traffic to the correct virtual interfaces. rules in the dom0 to forward traffic to the correct virtual interfaces.

To use network-route network-route, just uncomment the lines (network-scriptnetwork-route) (vif-scriptvif-route) and comment out the corresponding network-bridge network-bridge and and vif-bridge vif-bridge lines. That's really all there is to it. Restart lines. That's really all there is to it. Restart xend xend, and iptables iptables will show a new rule that handles forwarding to the will show a new rule that handles forwarding to the vif vif: #iptables-LFORWARD ChainFORWARD(policyACCEPT) targetprotoptsourcedestination ACCEPT0--192.168.42.190anywherePHYSDEVmatch--physdev-inn1 ACCEPTudp--anywhereanywherePHYSDEVmatch--physdev-inn1udpspt:bootpcdpt:bootps One common "gotcha" with network-route network-route is that it only works if the IP address is specified in the is that it only works if the IP address is specified in the vif= vif= line-otherwise the script doesn't know what rules to add. So, at minimum, the interface definition for the example above should look like this: line-otherwise the script doesn't know what rules to add. So, at minimum, the interface definition for the example above should look like this: vif=['ip="192.168.42.190",vifname="n1"']

Networking with network-bridge network-bridge is the currently accepted standard way of handling networking for guest domains in Xen. It uses the bridge tools in Linux to create a virtual layer 2 switch, to which it "plugs in" Xen virtual interfaces, as shown in is the currently accepted standard way of handling networking for guest domains in Xen. It uses the bridge tools in Linux to create a virtual layer 2 switch, to which it "plugs in" Xen virtual interfaces, as shown in Figure5-1 Figure5-1. It has the advantage of working with protocols that expect unadulterated Ethernet frames. This includes AoE, the pre-TCP/IP version of AppleTalk, NetBEUI, IPX, and a host of other protocols that date from the dark ages. It'll also work seamlessly with DHCP, which relies on broadcast packets. It's simple, logical, and robust.

The biggest disadvantage of network-bridge network-bridge (apart from its complexity) is that it tears down and rebuilds the real network interface when (apart from its complexity) is that it tears down and rebuilds the real network interface when xend xend starts. In some scenarios (for example, when dom0 has an NFS root) this can be unacceptable. This isn't a limitation of bridging, per se, only of attaching the bridge to the dom0 network device using the Xen scripts-if a dedicated physical device is used for the bridge, the problem goes away. starts. In some scenarios (for example, when dom0 has an NFS root) this can be unacceptable. This isn't a limitation of bridging, per se, only of attaching the bridge to the dom0 network device using the Xen scripts-if a dedicated physical device is used for the bridge, the problem goes away.

Another issue with network-bridge network-bridge is that it places the physical Ethernet device into promiscuous mode. This can be a tremendous resource hog on busy networks because the CPU has to intercept all packets rather than just those intended for its address. Finally, outgoing packets will have a different MAC address from the one on the physical card, which can lead to trouble with, for example, certain wireless networking hardware. This also throws many layer 2 http load balancers through a loop-anything that expects only one MAC address down each Ethernet port will be sorely confused. is that it places the physical Ethernet device into promiscuous mode. This can be a tremendous resource hog on busy networks because the CPU has to intercept all packets rather than just those intended for its address. Finally, outgoing packets will have a different MAC address from the one on the physical card, which can lead to trouble with, for example, certain wireless networking hardware. This also throws many layer 2 http load balancers through a loop-anything that expects only one MAC address down each Ethernet port will be sorely confused.

Figure5-1.Some of network-bridge's capabilities Even with these caveats, we recommend network-bridge network-bridge for most Xen servers. for most Xen servers.

You can start the network-bridge network-bridge script by hand if you like. For example, to manually create a bridge with the default name script by hand if you like. For example, to manually create a bridge with the default name xenbr0 xenbr0 attached to attached to eth0 eth0, type the following: /etc/xen/scripts/network-bridge{start stop}vifnum=0NoteIf you have vifnum vifnum set above set above nloopbacks nloopbacks, even if you only have one bridge, Linux will complain as if you had more bridges than loopbacks. This is because Xen uses the even if you only have one bridge, Linux will complain as if you had more bridges than loopbacks. This is because Xen uses the vifnum vifnum to determine the number of the virtual device it uses for the frontend, which presupposes the existence of the preceding virtual devices. Increase to determine the number of the virtual device it uses for the frontend, which presupposes the existence of the preceding virtual devices. Increase nloopbacks nloopbacks, and everyone is happy and everyone is happy.

network-bridge is the default networking option, thus is the default networking option, thus xend xend shouldn't need any configuration to use it. However, for completeness-to configure Xen to use shouldn't need any configuration to use it. However, for completeness-to configure Xen to use network-bridge network-bridge, modify xend-config.sxp xend-config.sxp to include the line to include the line (network-scriptnetwork-bridge)NoteOpenSUSE users might find that NetworkManager interferes with Xen's bridging. To fix this problem, go to YaST Network Devices Network Card YaST Network Devices Network Card and select the and select the Traditional Method with ifup Traditional Method with ifup option option.

This script causes Xen to use a bridge setup much like the following: #brctlshow bridgenamebridgeidSTPenabledinterfaces xenbr08000.feffffffffffnovif0.0 peth0 vif9.0 xenbr0 is, obviously, the name of the bridge. It bridges dom0's virtual Ethernet interface ( is, obviously, the name of the bridge. It bridges dom0's virtual Ethernet interface (vif0.0), the physical Ethernet card, and a domU's virtual interface. We can also see that STP (Spanning Tree Protocol) is disabled. In the default configuration, further domUs will simply have their interfaces added to this bridge.

STP is aimed at preventing loops in the network. You may want to turn STP on if you're doing anything complex with the virtual bridges. If you have multiple bridges and multiple network ports that you're using with Xen, it would probably be a good idea.

To rename the bridge, you can specify the bridge name as an option to the network-bridge network-bridge script: script: (network-script'network-bridgebridge=foo') Note also that network-bridge network-bridge defaults to binding defaults to binding eth0 eth0 to the bridge. To change the physical network card, use to the bridge. To change the physical network card, use (network-script'network-bridgebridge=foonetdev=eth1') Then the bridge setup becomes #brctlshow bridgenamebridgeidSTPenabledinterfaces foo8000.feffffffffffnovif0.01 peth01 vif9.0

Networking with network-nat network-nat is an extension of is an extension of network-route network-route that incorporates network address translation (NAT for short, or IP masquerade in some contexts). that incorporates network address translation (NAT for short, or IP masquerade in some contexts).

The network-nat network-nat script supplied with Xen works around script supplied with Xen works around network-route network-route's problem with DHCP in two ways. First, it can start a local DHCP server (so that guest domains can get addresses because they're now behind a router). If that's undesirable, it can create locally unique IP addresses using the domain ID. Then it sets up a standard iptables iptables rule for IP rule for IP masq masq: Dom0#iptables-tnat-n-L ChainPOSTROUTING(policyACCEPT) targetprotoptsourcedestination MASQUERADE0--0.0.0.0/00.0.0.0/0 Dom0#iptables-LFORWARD ChainFORWARD(policyACCEPT) targetprotoptsourcedestination ACCEPT0--10.0.0.0/16anywherePHYSDEV match--physdev-inn1 ACCEPTudp--anywhereanywherePHYSDEV match--physdev-inn1udpspt:bootpcdpt:bootps When a domain starts, it gets an IP address and adds appropriate iptables iptables rules for itself. Xen pa.s.ses the address to the kernel using the kernel-level IP autoconfiguration mechanism at boot (and isn't that a mouthful). rules for itself. Xen pa.s.ses the address to the kernel using the kernel-level IP autoconfiguration mechanism at boot (and isn't that a mouthful). network-nat network-nat can also integrate with your DHCP server. can also integrate with your DHCP server.

DomU#ifconfigeth010.0.2.1/16 DomU#routeadddefaultgw10.0.2.1 DomU#ping192.168.42.60 PING192.168.42.60(192.168.42.60)56(84)bytesofdata.

64bytesfrom192.168.42.60:icmp_seq=1ttl=63time=1.94ms This shows the default configured IP address for eth0 eth0 in domain 2. Actual numbers will vary depending on your setup, of course. in domain 2. Actual numbers will vary depending on your setup, of course.

Configuration Variables As we've mentioned, the two basic places where the administrator interacts with Xen's networking are in /etc/xen/xend-config.sxp /etc/xen/xend-config.sxp and in the domain config file. In each of these, you focus on one line: the and in the domain config file. In each of these, you focus on one line: the (network-script) (network-script) line in the former case and the line in the former case and the vif= vif= line in the latter. line in the latter.

Each of these will fail horribly, without explanation, over trivial errors in syntax. (This is Python, after all. You can put arbitrary Python code right in your config files, if you like. Configuration files should always be written in a language that's Turing-complete. Just ask Eric Allman.) The network-script network-script line is wrapped in parentheses, with arguments in quotes and separated by s.p.a.ces. For example, the following is valid: line is wrapped in parentheses, with arguments in quotes and separated by s.p.a.ces. For example, the following is valid: (network-script'network-bridgebridge=xenbr1') We've already discussed the vif= vif= line a bit. Note that the line a bit. Note that the vif vif configuration uses a completely different syntax from the network script setting, though: brackets with commas between arguments. configuration uses a completely different syntax from the network script setting, though: brackets with commas between arguments.

vif=['','bridge=xenbr1','bridge=xenbr2,ip="10.1.2.6"']

This line configures three interfaces, the first with default parameters, the second with a bridge argument, the third with bridge and IP. Note the commas between both separate interfaces and separate arguments.

Finally, in some examples you will see a dhcp=yes dhcp=yes line. The line. The dhcp= dhcp= line isn't necessary unless the kernel needs to get its address at boot-for example, if it's mounting its root filesystem over NFS. line isn't necessary unless the kernel needs to get its address at boot-for example, if it's mounting its root filesystem over NFS.

Custom Network Scripts You might be thinking at this point that it's overkill to specify a configuration script in the Xen config file rather than simply selecting among built-in options. Take our word for it: The ability to specify your own network script is fantastically useful. Because Xen's networking is built on standard tools, the scripts are easy to understand and tailor to your particular needs.

The easiest way to do this, and a sufficient method for most configurations, is to create a small wrapper script that calls the standard Xen scripts with appropriate arguments. You can also modify the standard scripts to source additional functions and call those as necessary-for example, to modify firewall rules or attach monitoring scripts.

Multiple-Bridge Setups Consider a scenario where you want inter-domU communication to occur on a purely virtual network, with a separate interface in each domain to communicate with the outside world.

In that case, you would create a pair of bridges, one with the default Xen setup, bridging the physical interface with the virtual ones, and one that bridges only virtual interfaces. Then you would specify both interfaces in the domain config file and configure them as normal from within the domain or the config file.

The wrapper would look something like this: #!/bin/sh dir=$(dirname"$0") "$dir/network-bridge""$@"vifnum=0 "$dir/network-bridge""$@"vifnum=1netdev=dummy0 This calls network-bridge network-bridge twice, the first time as normal and the second time with a twice, the first time as normal and the second time with a netdev netdev argument, causing argument, causing network-bridge network-bridge to use a dummy network device rather than a real one. to use a dummy network device rather than a real one.

To tell xend xend to run this script at startup, change the to run this script at startup, change the network-script network-script line in line in xend-config.sxp xend-config.sxp as follows: as follows: (network-scriptmy-wrapper) Make sure that the my-wrapper my-wrapper script is executable, or else nothing will work. script is executable, or else nothing will work.

To use these bridges from the domUs, specify the correct bridge in the vif= vif= line: line: vif=['mac="aa:0:1:2:3:4",bridge="xenbr1"']

Bridged and Routing A slight modification to this scenario puts the domU communication on its own bridge, which is then routed via iptables iptables rules in the dom0, as shown in rules in the dom0, as shown in Figure5-2 Figure5-2. (Arjen Runsink, who wrote a script that does this, calls this a brouter brouter-a portmanteau of bridge and router.) Figure5-2.Combining bridging and routing This creates a standard bridge, but it doesn't attach the physical device to it. Instead the bridge gets an IP address and a route. When a domU starts, its vif vif is attached to the bridge by the ordinary is attached to the bridge by the ordinary vif-bridge vif-bridge script. script.