In this article we will take a look at networking in Xen. Each domain network interface is connected to a virtual network interface in dom0 by a point to point link. These devices are named vif<domain id><interface id> (e.g. ‘vif1.0’ for the first interface ‘eth0’ in domain 1).
Traffic on these virtual interfaces is handled in dom0 using standard Linux mechanisms for bridging, routing with NAT or two-way routing. Typically, only one of these mechanisms can be used at one time. These mechanisms are defined in shell scripts under the directory ‘/etc/xen/scripts/’ and can be used by xend to perform initial configuration of the network and configuration of new virtual interfaces. The default mechanism that is used by xend is bridging. You can change this by commenting out the bridging scripts and enabling your preferred mechanism’s scripts in the file ‘/etc/xen/xend-config.sxp’. Each mechanism has two related scripts as shown below:
- Bridge Networking: (network-script network-bridge), (vif-script vif-bridge).
- Routed Networking with NAT: (network-script network-nat), (vif-script vif-nat).
- Two-way Routed Networking: (network-script network-route), (vif-script vif-route).
1. Bridging
The default Xen configuration uses bridging within Domain0 to allow all domains to appear on the network as individual hosts. When xend starts, the physical interface ‘eth0’ is brought down. Then, a virtual network bridge ‘veth0’ is created and renamed ‘eth0’. Finally, the physical interface is brought up and renamed to ‘peth0’ and is connected in the bridge ‘eth0’. Afterwards, when a new domain is started its virtual interfaces (e.g. vif1.0) are connected in the ‘eth0’ bridge (Figure 1).
Figure 1 – Connecting all the network interfaces on a virtual network bridge
In this case the ‘/etc/network/interfaces’ file on the dom0 will look like this:
eth0 Link encap:Ethernet HWaddr 00:11:d8:55:4d:74
inet addr:192.168.1.100 Bcast:192.168.1.255
Mask:255.255.255.0
inet6 addr: fe80::211:d8ff:fe55:4d74/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500
Metric:1
RX packets:58420 errors:0 dropped:0 overruns:0 frame:0
TX packets:34142 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:27060374 (25.8 MiB) TX bytes:4589683 (4.3 MiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:89 errors:0 dropped:0 overruns:0 frame:0
TX packets:89 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:8124 (7.9 KiB) TX bytes:8124 (7.9 KiB)
peth0 Link encap:Ethernet HWaddr 00:11:d8:55:4d:74
inet6 addr: fe80::211:d8ff:fe55:4d74/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST
MTU:1500 Metric:1
RX packets:65661 errors:0 dropped:0 overruns:0 frame:0
TX packets:34023 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:29744006 (28.3 MiB) TX bytes:4583617 (4.3 MiB)
Interrupt:17
Vif1.0 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST
MTU:1500 Metric:1
RX packets:6 errors:0 dropped:0 overruns:0 frame:0
TX packets:24 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:32
RX bytes:362 (362.0 B) TX bytes:4739 (4.6 KiB)
Vif2.0 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff
inet6 addr: fe80::fcff:ffff:feef:ffff/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST
MTU:1500 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:32
RX bytes:368 (382.0 B) TX bytes:4934 (4.8 KiB)
1.1. A basic network configuration
The simplest configuration is to use bridging in order for the domU guests to join our existing network. Let’s suppose that our home network is 192.168.1.0/24 with a gateway with IP address 192.168.1.1. The primary network interface in our dom0 system has a static IP address assigned which is 192.168.1.100. We can grab a new IP from this network (e.g. 192.168.1.101) and assign it to our new domU guest. By doing this, our domU guest will be fully visible and available on our existing network, allowing all traffic in both directions.
To achieve this setup we must perform the following steps:
1. Edit the xend configuration file ‘/etc/xen/xend-config.sxp’ and enable these options: (network-script network-bridge) (vif-script vif-bridge) if they are not already enabled.
2. Edit the domU guest’s ‘/etc/network/interface’:
1. Edit the xend configuration file ‘/etc/xen/xend-config.sxp’ and enable these options: (network-script network-bridge) (vif-script vif-bridge) if they are not already enabled.
2. Edit the domU guest’s ‘/etc/network/interface’:
root@dom0$ mount –o loop /home/giorgos/xen/domu1.img /mnt/
root@dom0$ vi /mnt/etc/network/interfaces
root@dom0$ umount /mnt
If we prefer the domU guest’s primary network interface to take an IP address from a DHCP server we must add the following lines to the above file:
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet dhcp
Otherwise, if we would like to give a static IP address (e.g. 192.168.1.101) to the domU guest’s primary network interface we have to add the following lines:
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.1.101
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1
In the domain’s configuration file (e.g. ‘/etc/xen/domu1’) we must make sure that we have enabled the option vif = [ ‘’ ]. Additionally, if we want to use DHCP we must make sure that we have enabled the option dhcp= “dhcp”.
1.2. Custom Bridging
If we cannot use IP addresses of the physical network that our host system is connected, the ideal solution is to create a new network for our domU guests. Thus, let’s say that our host system is connected on 192.168.1.0/24 network via the interface ‘eth0’. The main idea behind this approach is to create a custom virtual bridge and assign it an IP address of a different network. Then all the domU guests will connect to this newly created network having the dom0’s bridge as their gateway. However, in order for the domU guests to reach the physical network 192.168.1.0/24 where the dom0 is connected in, some routing must be done on the host between the virtual bridge and its primary network interface. Figure 2 summarizes this setup.
Figure 2 – Connecting the domU guests on a different network.
For this setup to take effect we have to disable the option (network-script network-bridge) in the ‘/etc/xen/xend-config.sxp’ configuration file. On the other hand, the option (vif-script vif-bridge) must be left enabled.
Let’s assume that we want to use the network 192.168.2.0/24 to connect our domU guests. We will assign to the bridge the IP address 192.168.2.1 as shown in figure 2. To create the new bridge we modify the ‘/etc/network/interfaces’ file on dom0 as follows:
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.1.100
netmask 255.255.255.0
network 192.168.1.0
broadcast 192.168.1.255
gateway 192.168.1.1
# The bridge interface
auto br0
iface br0 inet static
address 192.168.2.1
netmask 255.255.255.0
bridge_ports none
bridge_maxage 12
bridge_stp off
bridge_fd 1
bridge_hello 2
Finally, for the above configuration to work, we must make some additional routing configurations on dom0. At this point, the domU guest will be able to communicate properly with dom0 and vice versa. However, in order for the domU guest to communicate with other machines on 192.168.1.0/24 network or with the Internet we must enable routing on dom0. To do this we run the following command as root:
root@dom0$ echo 1 > /proc/sys/net/ipv4/ip_forward
In order to make this change permanent we can set the option ‘net.ipv4.ip_forward=1’ in the ‘/etc/sysctl.conf’ file. A final step is to enable IP masquerading with the help of IP tables. To do this, we create a new file: ‘/etc/network/if-up.d/dom0-routing’ with the following content:
#!/bin/sh
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
exit 0
After that, we must set correct permission to this file by running:
root@dom0$ chmod 755 /etc/network/if-up.d/dom0-routing
After completing the above steps the network configuration needed to provide network connectivity to the domU guests is ready. Now, we can restart the dom0 system. When the system restarts we can run:
root@dom0$ ifconfig
The output must be like this:
br0 Link encap:Ethernet HWaddr fe:ff:ff:ff:ff:ff
inet addr:192.168.2.1 Bcast:192.168.2.255
Mask:255.255.255.0
Mask:255.255.255.0
inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500
Metric:1
Metric:1
RX packets:17 errors:0 dropped:0 overruns:0 frame:0
TX packets:81 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:972 (972.0 B) TX bytes:8599 (8.3 KiB)
eth0 Link encap:Ethernet HWaddr 00:11:d8:55:4d:74
inet addr:192.168.1.100 Bcast:192.168.1.255
Mask:255.255.255.0
Mask:255.255.255.0
inet6 addr: fe80::211:d8ff:fe55:4d74/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500
Metric:1
Metric:1
RX packets:4703 errors:0 dropped:0 overruns:0 frame:0
TX packets:2651 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1044816 (1020.3 KiB) TX bytes:406619 (397.0
KiB)
KiB)
Interrupt:17
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:89 errors:0 dropped:0 overruns:0 frame:0
TX packets:89 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:8124 (7.9 KiB) TX bytes:8124 (7.9 KiB)
Also, we can run on dom0:
root@dom0$ route –n
We will get:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 br0
Note the final line in the above example. This line specifies the route to the newly created network 192.168.2.0/24. All the packets will be routed to the bridge ‘br0’ in order to reach the domU guests. Then, the bridge will route each packet to the correct virtual interface which is connected to it.
It’s time to do some configuration on our domU guest:
root@dom0$ mount –o loop /home/giorgos/xen/domu1.img /mnt
root@dom0$ chroot /mnt
root@dom0$ vi /etc/network/interfaces
root@dom0$ exit
root@dom0$ umount /mnt
In the guest’s ‘/etc/network interfaces’ we put the following lines:
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address 192.168.2.2
netmask 255.255.255.0
network 192.168.2.0
broadcast 192.168.2.255
gateway 192.168.2.1
up route add -net 192.168.2.0/24 gw 192.168.2.1
up route add default gw 192.168.2.1
As shown in the above example, we configure the primary network interface of the domU guest ‘eth0’ with the IP address 192.168.2.2. Also, we specify the netmask, the network, the broadcasting address and finally the gateway 192.168.2.1. The final two lines need some more explanation.
As we said earlier the dom0 guest is connected on the physical network 192.168.1.0/24 whereas the domU guest is connected on the network 192.168.2.0/24. In order for the domU guest to be able to access the machines on the network 192.168.1.0/24 and vice versa we must perform some routing. With the line ‘up route add -net 192.168.1.0/24 gw 192.168.2.1’ we add a new route to the network 192.168.0/24 specifying that all the packets that are directed for that network must be send on dom0’s end which is 192.168.2.1. Then, dom0 is responsible to properly route this packets. The final line ‘up route add default gw 192.168.2.1’ adds a default route with gateway the dom0’s bridge which has the address 192.168.2.1. This helps the domU guest to communicate with other hosts on the Internet.
Finally, for the above configuration to work, we must make some modifications on the domain’s configuration file (e.g. ‘/etc/xen/domu1’). We must make sure that we have enabled the option vif = [ ‘’,‘bridge=br0’ ]. With this option we specify the custom bridge ‘br0’ to xend. Thus, every time a new domain is started, its virtual interfaces (e.g. vif1.0) are connected in the ‘br0’ bridge automatically by the ‘vif-bridge’ script.
It’s time to test our configuration. Let’s run the domU guest:
root@dom0$ xm create –c domu1
After the guest boots we can run the following command to see if the primary network interface is correctly configured:
root@domu1$ ifconfig
The result must look like this:
eth0 Link encap:Ethernet HWaddr ce:15:d1:2d:86:f0
inet addr:192.168.2.2 Bcast:192.168.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:57 errors:0 dropped:32 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:3121 (3.0 KiB) TX bytes:159 (159.0 B)
Interrupt:5
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Also, in order to see if the routing is properly configured we can run:
root@domu1$ route –n
The result must look like the lines below:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.2.1 0.0.0.0 UG 0 0 0 eth0
192.168.1.0 192.168.2.1 255.255.255.0 UG 0 0 0 eth0
192.168.2.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
We can try to ping the dom0 and some Internet addresses to see if everything works properly. Also, we can try to ping the domU guest from dom0.
We have now configured a domU guest with network access. We can easily configure more domU guests to have network access by assign them IP addresses from the range of 192.168.2.0/24.
In routed networking with NAT (Network Address Translation) a private LAN is created for Xen guests. Traffic coming from the guests is then networked to the outside network via NAT. In this case, dom0 will automatically perform all the NAT’ing required.
Let’s suppose that our home network is 192.168.1.0/24 with a gateway with IP address 192.168.1.1. The primary network interface in our dom0 system has a static IP address assigned which is 192.168.1.100. We will create a virtual private LAN for domU guests (e.g. 192.168.2.0/24). The domU guests must NAT via dom0, which will be the gateway of the new virtual network, to reach the home LAN. Thus, traffic on the home LAN appears as if coming from dom0 (192.168.1.100). The domU guests can be directly accessed from 192.168.1.0/24, however a route must be added to the gateway (192.168.1.1) for this to happen. To achieve this setup we must follow these steps:
1. Edit the xend configuration file ‘/etc/xen/xend-config.sxp’ and enable these options (network-script network-nat) (vif-script vif-nat). Also, we must disable the other networking scripts.
2. Edit the domU guest’s ‘/etc/network/interface’:
1. Edit the xend configuration file ‘/etc/xen/xend-config.sxp’ and enable these options (network-script network-nat) (vif-script vif-nat). Also, we must disable the other networking scripts.
2. Edit the domU guest’s ‘/etc/network/interface’:
root@dom0$ mount –o loop /home/giorgos/xen/domu1.img /mnt/
root@dom0$ vi /mnt/etc/network/interfaces
root@dom0$ umount /mnt
We will give a static IP address (e.g. 192.168.2.1) to the domU guest’s primary network interface and will assign its gateway to 192.168.2.254 which will be the host end:
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.2.1
netmask 255.255.255.0
network 192.168.2.0
broadcast 192.168.2.255
gateway 192.168.2.254
3. In the domain’s configuration file (e.g. ‘/etc/xen/domu1’) we must make sure that we have enabled the option vif = [ ‘’ ].
3. Two-way Routed Networking
In this setup forwarding rules must be put in manually. This setup however allows for the greatest flexibility when it comes to routing and setting up a private network. Routing creates a point-to-point link between dom0 and each domU. Routes to each domU are added to dom0’s routing table, so domU must have a static IP. DHCP doesn’t work in this setup.
Let’s suppose again that our home network is 192.168.1.0/24 with a gateway with IP address 192.168.1.1. The primary network interface in our dom0 guest has a static IP address assigned which is 192.168.1.100. We will create a private LAN for domU guests (e.g. 192.168.2.0/24). The domU traffic is routed to the home network with the help of dom0. The domU guests can be directly accessed from 192.168.1.0/24, however a route must be added to the gateway (192.168.1.1) for this to happen. To achieve this setup we must follow these steps:
1. Edit the xend configuration file ‘/etc/xen/xend-config.sxp’ and enable these options (network-script network-route) (vif-script vif-route). Also we must disable the other networking scripts.
2. Edit the domU guest’s ‘/etc/network/interface’:
1. Edit the xend configuration file ‘/etc/xen/xend-config.sxp’ and enable these options (network-script network-route) (vif-script vif-route). Also we must disable the other networking scripts.
2. Edit the domU guest’s ‘/etc/network/interface’:
root@dom0$ mount –o loop /home/giorgos/xen/domu1.img /mnt/
root@dom0$ vi /mnt/etc/network/interfaces
root@dom0$ umount /mnt
We will give a static IP address (e.g. 192.168.2.1) to the domU guest’s primary network interface and will assign its gateway to 192.168.2.254 which will be the host end:
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.2.1
netmask 255.255.255.0
network 192.168.2.0
broadcast 192.168.2.255
gateway 192.168.2.254
3. In the domain’s configuration file (e.g. ‘/etc/xen/domu1’) we must make sure that we have enabled the option vif = [ ‘’ ].
4. We must also configure dom0 for routing. To achieve this we run the following command as
root:
root@dom0$ echo 1 > /proc/sys/net/ipv4/ip_forward
#!/bin/sh
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
exit 0
root@dom0$ chmod 755 /etc/network/if-up.d/dom0-routing
In order for the machines on the network 192.168.1.0/24 to be able to see the network 192.168.2.0/24, we need to add a route. This is added into the Default Gateway of the external network (192.168.1.1) so that when any machine queries a 192.168.2.0/24 address, their default gateway is checked for an entry.
- David Chisnall: The Definitive Guide to the Xen Hypervisor. Prentice Hall, 2007
- XenParaVirtOps [http://wiki.xensource.com/xenwiki/XenParavirtOps#head-6a64160b847b6fbd780350181436269c5a0a5b11]
- Xen Networking [http://wiki.xensource.com/xenwiki/XenNetworking]
- Bridging Network Connections [http://wiki.debian.org/BridgeNetworkConnections]
- The Linux Kernel Archives [http://www.kernel.org/]
------
Get the full pdf here: http://www.cs.uoi.gr/~gkappes/tutorials/xen_guide.pdf
No comments:
Post a Comment