This page describes an example Xen and Xen Linux deployment at an Internet datacentre in London. Two machines are located in the same rack within the Redbus-2 datacentre. They provide a number of services to different organisation including XenSource, the University of Cambridge, and private individuals.

Networking

Each machine has an Ethernet connection to the ISP's switch. Additionally the machines are connected together via a crossover cable between their second gigabit Ethernet ports. A software bridge is created for each physical interface. The crossover link and the bridges at either end of it form an effective internal network.

RIPE, the European IP address registry, are very keen to conserve public IP addresses. Therefore we only use a small number of public addresses. Each machine has a public address for domain 0 and we have two additional public addresses for guest domains. Other guests use private addresses and connect to the bridge connected to the internal network. The domain 0s provide IP masquerading for outgoing connections, and port forwarding for selected incoming ports.

We previously performed application-level (reverse) proxying in domain 0, e.g. Apache in proxy mode or Exim in hubbed-hosts mode. We no longer use this technique as it causes trouble with network buffer allocation for dom0-domU traffic (buffers are held on to for too long). It is also not recommended to use dom0 for anyting other than domain management.

Guest domains do not necessarily have to use the domain 0 of their own physical machine for either incoming or outgoing proxying/masquerading - the topology just looks like a single large network with two possible gateways.

Storage

The current storage configuration (likely to change in the future to something more migration-friendly) is as follows. Each machine has two physical disk partitions of interest, one for its own domain 0 and one for LVM for the guest domains. The LVM allocation (in terms of logical volume names and sizes but not necesarily layout) is the same on each machine. Nightly cron jobs create an LVM snapshot of a running domain's volumes, these are then dd'ed to the same volume on the other machine. The snapshot is then deleted. Off-site backup is performed using rsync on live filesystems.

Software

The machines use xen-2.0.5 and Linux 2.6.11-xen. All domains currently use Debian sarge.

Application-layer (reverse) proxying is performed by some domUs for other domUs which have private addresses. These are currently:

Apache using name-based virtual servers running in proxy-passthrough mode:

<VirtualHost *>
    ServerName lists.xensource.com
    ProxyPass / http://192.168.0.35/
</VirtualHost>

Exim4 providing forwarding for "hubbed hosts"