Contents

Bernhard Kuhn
1. Introduction
2. Does it Work?
DavidMuench
ChadKitching
PaulTap
MarkHurenkamp

Bernhard Kuhn

Introduction

This is a multiseat setup where a PS2 Keyboard+Mouse and the first of two nVidia 7800GTX graphics cards belong to the xen0 domain and the on-board USB host controller and the second nVidia 7800GTX card are dedicated to a xenU domain. It's possible to play UT2004 simultaneously

I wanted to create a multi-seat environment with two 7800GTX PCIe cards, two PCI sound cards, two mice and two keyboards, so that two people can work simultaneously on the same computer using different operating systems (i.e. Linux, BSD, Solaris or potentially even some flavour of Windows - though this is not yet possible without virtualization support by CPU, so my current experiments are Linux-only).

What i did:

Installed Fedora Core 5 on physical computer
Installed Fedora Core 5 on virtual computer (using qemu).
Installed Xen-3.0.2 from source on the physical computer.
Installed the unified xen0/xenU kernel on the virtual computer
Configured gdm on the virtual computer to use "-shartvt vt1"
Installed an second PCI USB controller card, isolated it from the xen0 domain and dedicated it to the xenU domain.

Does it Work?

This basically works, that is, the physical computer boots linux and the X-server comes up on the first 7800GTX. Then i log in and start then virtual computer (xm create ...) and another X-server comes up on the second 7800GTX showing the gdm login window.

Now here's the problem: The second USB mouse works great, but for the second USB keyboard it is only possible to do one or two keystrokes, then the keyboard is dead. When rebooting the virtual machine, then i can again type one or two letters before the keyboard stops operations.

Another observation is that the physical computer got pretty unstable after adding the PCI USB controller - so i guess that hardware is broken?

For additional details on how to get the nvidia 3D driver working with xen, please check

Update: instead of using a second USB controller, i added a PS2 mouse and a PS2 keyboard for the xen0 domain and used the the onboard USB controller for the xenU domain exclusively.

It's now possible to play UT2004 at the same time at both consoles (xen0 and xenu) with decent frame rates (30-50 fps).

However, there are two issues left:

i can't get german keyboard layout working (i guess it has something to do with the mandatory "evdev" driver).
When shutting down the xenU domain, then the system gets stuck (mouse pointer on the X-Server in xen0 still moves) with a spinlock bug (i'll try to capture the kernel log and post it at a later time).

DavidMuench

Note

Note: I have since abandoned using MythTV in Xen - my Myth backend was just consuming too many resources, and I decided moving it to a separate box was the wisest solution.

Introduction

I'm using Xen to consolidate servers in my house. Prior to Xen I had a number of old, somewhat reliable machines all doing their own thing and consuming a lot of power. I've succeeded in consolidating all of those machines into one Xen machine, plus a file server and a firewall machine. The applications I am running include Apache (with PHP and all that), MySQL, the MythTV backend, and the Asterisk PBX. I'm running Ubuntu, Gentoo, and CentOS in various domains. As of this writing I am using Xen 2.0.6, except for in the MythTV domain which is running a 2.0-testing kernel because of a bugfix the IVTV driver requires.

Hardware Config

My hardware configuration currently consists of an Intel 865 chipset motherboard with a 2.4ghz Celeron D processor and 1.5 gig of memory in a Supermicro 4U rack mount case. Two Samsung 160 gig IDE drives are in a RAID-1 configuration using Linux's MD software raid. There is an onboard gigabit ethernet port which is used as the main network connection for dom0 and all but one of the domU's. The external webserver domU has it's own dedicated 100mbit PCI network card in order to put it on a DMZ lan. Also there are 3 Happauge PVR-250 MPEG-2 encoder cards, 1 pcHDTV-3000 HDTV card, and a generic 4 port USB PCI card.

Domain 0

Domain 0 is running Gentoo using the Xen ebuilds that are currently available in Gentoo's bugzilla (and will hopefully be in Gentoo proper soon). Beyond Xen itself, it doesn't run much other than thttpd in order to serve up statistics about the Xen installation. EVMS is used to manage the disks. All of the PCI cards mentioned above (3 PVR-250's, the pcHDTV, PCI network card, and USB card) are hidden from dom0.

Filesystems and Volume Management

Each domU has a 3 to 5 gig root volume (/dev/evms/hostname-root) and a 512 meg swap volume (/dev/evms/hostname-swap) provided by EVMS (using the LVM2 plugin). The external webserver domU also has a 4 gig volume for storing our photo album in our web gallery, and the MythTV backend domU has a 16 gig volume to buffer live TV. That live TV buffer volume also uses the XFS filesystem, all other filesystems in all domains use ext3.

Other Domains

Network management: Running Ubuntu 5.04, 64 meg of ram. Runs DHCP, MyDNS, a TFTP server, and other miscellaneous small programs.
Database server: Running Ubuntu 5.04, 128 meg of ram. Runs MySQL and provides database services to everything else on my network.
External web server: Running Ubuntu 5.04, 128 meg of ram. Runs Apache 2 with PHP 4 and serves our family's web site plus a couple others. It has a reasonably large Gallery installation on it. This domain also has the extra PCI network card assigned to it and does not have a virtual ethernet interface connected to Domain 0. This allows me to plug this interface into my firewall's DMZ and keep things secure.
Internal web server: Running Ubuntu 5.04, 96 meg of ram. Runs our "intranet", such as the MythTV web interface, phpMyAdmin, a recipe database, the MyDNS web interface, and hosts a small Subversion repository to keep track of my scripts and code.
Asterisk PBX: Running CentOS 4.0, 128 meg of ram. The Asterisk PBX serves the VOIP phones in our house and routes them to our VOIP provider. It also provides voicemail and many other features. I am running a manually installed copy of Asterisk@Home currently.
MythTV backend: Running Gentoo, 256 meg of ram. This domain runs the 0.18-fixes branch of MythTV and talks to our two MythTV frontend machines in the house. It has the HDTV card, the three PVR-250 cards, and the PCI USB card assigned to it. The HDTV card is not currently in use and I have no idea yet if it works under Xen. But the three PVR-250 cards do work very well, even when all three are recording at once. The PVR-250 cards are using the IVTV driver version 0.2.0-rc3k. My satellite provider is DirecTV, and I have 3 recievers for my 3 PVR-250 cards. Each one is controlled (channel changes) by a serial line consisting of a USB to serial adapter (using the pl2303 driver) connected to a home made DB9 to telephone handset jack cable. There are instructions on the net on how to build this cable. Since Xen 2.0 doesn't virtualize USB, I use a separate USB PCI card and assigned that to the MythTV domain to work around that problem. There is a 16 gig XFS filesystem (deletes of large files on ext3 take too long) to buffer live TV, and the scheduled recordings are stored on a NFS mount from my fileserver.

IVTV Issues

In order for IVTV to work in a domU as of this writing, the domain must use Xen 2.0-testing and the IVTV driver needs a small tweak to drivers/ivtv-driver.c on like 675 as shown below:

        } else if (pci_bus != NULL)
                IVTV_KERN_INFO("XXX PCI device: 0x%04x vendor: 0x%04x\n",
                                pci_bus->device, pci_bus->vendor);

The stock IVTV code is missing the "if (pci_bus != NULL)" part of the statement, and will cause modprobe to crash when inserting the driver. Hopefully this will be rolled into the official IVTV version soon.

Does it Work?

I was pretty skeptical when I started this project months ago. I never figured I'd get all of this running reliably, much less on a single 2.4ghz Celeron. There have been many problems to overcome, and I owe a lot to the people on the xen-devel mailing list. But it is running reliably, even when recording 3 simulaneous programs in MythTV, playing programs back to my two frontends, making phone calls with Asterisk, and people hitting the web servers - all at once. The most CPU intensive task is probably commercial flagging recorded programs in MythTV, and that doesn't even bother the machine. I still have 203 meg of memory unallocated too.. Hmm, what else can I do...

ChadKitching

Introduction

I had been running a Linux firewall/router machine for a while, and was disappointed with the fact I couldn't really use the extra horsepower and storage capacity of the machine. Using Xen let me partition off a small segment of the disk to dedicate to the firewall, and use the rest of the memory and CPU power to run other tasks.

Hardware Config

The machine is fairly low powered. It runs a Pentium 4 1.80 GHz with 256MB of RAM and an 80GB disk. It has two NICs, one built-in, and a 3COM 3C905 PCI card. It's also equipped with a PVR-150 card. Nothing terribly fancy here.

Domain 0

My domain 0 is currently running Ubuntu 5.10 with the Xen 2.07 debs from Xenophilia. I have 128MB of RAM dedicated to this domain. I'm running MythTV in this domain along with the PVR-150. I hope to move this into a DomU some day, but since Myth, X and the PVR-150 card need so much memory, and the fact that X doesn't correctly work in domains other than Dom0 means that it was safer to run everything in this domain. I'm running the latest SVN version of the IVTV driver as of this writing (pre-0.52, December 28, 2005), and it works without problem. The only thing to remember is that you must build it with "ARCH=xen" on the make command line.

Network configuration

I have two network cards visible in domain 0. One is for the Internet link to my ISP, and the other is for my LAN. To make it easier to identify which is which, I use ifrename and an /etc/iftab to rename my interfaces red and grn to correspond to the card designations in IPCop. The red network interface is completely firewalled from dom0, but since it uses PPPoE to connect to my ISP, the danger is pretty low anyway.

This is the /etc/network/interfaces that I use to bring up my network cards:

auto xen-br0 xen-br1
iface xen-br0 inet static
        address 10.241.1.25
        netmask 255.255.255.0
        gateway 10.241.1.1
        dns-nameservers 10.241.1.1
        bridge_ports grn
iface xen-br1 inet manual
        bridge_ports red
        post-up /sbin/iptables -A INPUT -i red -j DROP
        post-up /sbin/iptables -A INPUT -i xen-br1 -j DROP

IPCop domain U

This domain was more difficult to get working. IPCop uses the 2.4 kernel patched with OpenSwan, and a particular version of the iptables patch-o-matic. Installation of this domain required running it on a separate machine (or through a complete virtual machine like qemu), tarring up the filesystem, and scping the resulting tarball to the Xen machine. Make sure you configure the correct number of NICs during installation, because the IPCop setup program isn't terribly fond of ethernet interfaces that don't require the ethernet modules it knows about. If you mess up the setup, you can manually specify interfaces in /var/ipcop/ethernet/settings, but make sure you set a driver for each one.

Building the custom kernel

I chose to build a kernel without hardware access because I wasn't confident in the state of the 2.4 kernel in Xen. I downloaded IPCop, and linux-2.4.31 and applied the patches for patch-o-matic, OpenSwan, and Xen. I had to modify the arch/xen/config.in to allow me to select CONFIG_NETDEVICES even when CONFIG_XEN_PHYSDEV_ACCESS was turned off in order to compile the ppp and pppoe modules. Otherwise the unprivileged domain was unable to connect to my ISP. The IPCop .config file required a few modifications (enabling xen and the front-end drivers and turning off the physical ethernet drivers that would cause the compile to fail because of the config.in hack) and a few makes (oldconfig, dep, etc) later I had a kernel suitable for this task.

Modifying IPCop for Xen

Few changes are needed for IPCop to work properly. You'll obviously need to transfer the modules from the custom kernel to the IPCop filesystem, but I'd also suggest removing the /sbin/hwclock before booting the domain. hwclock doesn't work in domU, and causes the boot process to pause for a long time.

Xen config file

This is the configuration file I use for my IPCop domain. IPCop installs with hda1 or sda1 being the /boot partition, hda2/sda1 being /var and hda4/sda4 being the root filesystem. Make sure you either honour this, or alter the /etc/fstab of the machine to match your preferred filesystem. IPCop normally uses a swapfile instead of a swap partition, which is likely why hda3 is unused.

kernel = "/boot/xen/domU/vmlinuz-2.4.31-ipcop-xenU"
memory = 32
name = "ipcop"
disk = [ 'phy:vg1/ipcop_boot,hda1,w',
         'phy:vg1/ipcop_var,hda2,w',
         'phy:vg1/ipcop_root,hda4,w' ]
root = "/dev/hda4 ro quiet"
nics = 2
vif = [ 'mac=5A:12:34:56:78:9A, bridge=xen-br0',
        'mac=5A:FE:DC:BA:98:76, bridge=xen-br1' ]

Does it work?

So far, I'm quite happy with how it works. I'm able to log into my PPPoE DSL with it without any problems, and the IPSec VPN works great (well, same as it works on a dedicated machine). Even while using Myth on the machine, there seems to be no internet or VPN slowdown. When I get a little more memory, I hope to separate the Mythbackend/cards and fileserver into their own domains, but until that happens, this works well.

PaulTap

Introduction

I'm running Xen (version 2.0.6 still) on an IBM Bladecenter plus Coraid ATA over Ethernet storage. The idea is to offer Virtual servers with a rather "simple" look and feel to the application users whereas typical hardware related issues like redundancy are dealed with in the Xen layer. All physical servers are equiped with 4 NIC's of which 2 are used for the Xen-U domains and 2 are used for the Xen-0 domains. The AoE storage is only available in the Xen-0 domains and is offered to the Xen-U domains using LVM.

Xen-U domains

One of the Xen-U domains runs as my Desktop, having a Fedora 4 plus NX Server installed. It has been running for over a year now without any problems. Another domain acts as a mailserver, and I've had some others up for testing. Due to things in the To-do list, full deployment has not yet taken place.

To-do

- perform fail over on the physical NICs - boot the Xen-0 domains PXE/AoE - upgrade to Xen 3 (has been tested already with Fedora 5)

Coolest Demo

Migration of the Desktop Dom-U from one blade to another while using an NX session to conect via ssh to the Dom-0 to issue the migration command. This all worked without losing the NX session, allthough it isn't 100% fail-safe so far.

MarkHurenkamp

Introduction

I've been playing with Xen ever since the 1.2 release, and was very impressed with the features of the 2.x releases. So much that when my server was due for an upgrade, I decided to split it up into several Xen instances. Here's my setup:

Hardware:

Already running for a few years now, this is a cheap AMD duron platform, only a single 1.2Ghz CPU, but with a lot of Ram (2Gb). For telephony (and as a backup for ADSL failures) it has a Fritz ISDN pci card, and for multimedia purposes it has a hauppauge PVR500 dual tuner card. Besides it's onboard 100Mbit ethernet, it has 3 more network devices.

Dom0: Host running Ubuntu 7.10

Although I started out with a xen host / firewall dom0, with the upgrade to kernel 2.6.22.x (2.6.18 as well) and shorewall 3.4.x I could not get that combination to work, and so I've moved the firewall to a domU.

This machine runs a Xen snapshot of 3.1.4, and the accompanied 2.6.18.x kernel, and creates two xen bridges: one for the DMZ and one for Local machines.

DomU: Firewall using network card; running Ubuntu 7.10 with Shorewall

This machine functions primarily as a gateway between my other machines and the internet, I gave it direct access to the ethernet card which is connected to the net, so that it has best access to networking hardware. Local and DMZ domains are connected via a bridge to dom0.

DomU: Mediaserver using PVR500; running Ubuntu 7.10 with Mythtv & Apache

This is a virtual machine that I added recently, since I had built a MythTV based PVR, but wanted to offload the recording and datagathering parts, I moved the backend to my server. It took some time to get it to cooperate properly with IVTV, but it works well now. A recent upgrade to Ubuntu 7.10 made the system even more stable, plus it works out of the box with the ubuntu packaged kernel 2.6.22.x ivtv driver. Being a mediaserver it obviously uses most of the now almost 1Tb of diskspace that this machine has, and although at first I thought that 2 tuners would not be enough for my needs, I find that I now have more than plenty of programs to choose from whenever I want to watch TV. And using the mythweb plugin, I can now program the mediaserver using a nice webinterface.

DomU: PBX using Fritzcard; running Ubuntu 7.10 with Asterisk

Having been assigned 2 VOIP numbers by my ISP (xs4all.nl), I decided to install my own PBX, so that I could seamlessly integrate VOIP, ISDN, and my existing analog phones (for which I used a Linksys PAP2T). Using my fritzcard as an interface between Asterisk and my external ISDN line. Nice thing is that with my wifi enabled mobile phone (Nokia N95), I can now pick up any incomming line when I'm at home (currently running ubuntu 7.10)

DomU: Mailserver; running Gentoo Linux with QMail & Vpopmail

This is a very basic system, I used to run mail, web and firewall together, but since that often caused problems when upgrading of one of these services was needed, I decided to seperate them. Running QMail since it is one of the best/safest mail programs around, and vpopmail so that I don't need to give mail users accounts on the machine.

Yet to be upgraded to Ubuntu 7.10.

DomU: Webserver; running Gentoo Linux with Apache & MySQL

This is one of the main functions of the system, it needs to provide my web pages (hoth.xs4all.nl/~hurenkam), although the website can use a redesign, I still use it often to post mostly photos for family & friends, but also articles every now and then which I think may be usefull for others.

Yet to be upgraded to Ubuntu 7.10.

DomU: Fileserver; running Gentoo Linux with samba

This system exports the scattered nfs filesystems (from media server, webserver, mailserver) and re-exports them via samba so that I have easy access from windows and mac systems. Yet to be upgraded to Ubuntu 7.10.

Performance

I guess given the amount of services the machine runs, I am surprised that it runs as well as it does. I rarely reboot the machine, and even mythtv remains stable over long periods of time (several months). I did disable any transcoding or commercial flagging jobs on the mediaserver, since it seems to slow it so much that it interfered with other recording jobs. Also need to mention here that my telephone traffic is very low, and thus the PBX does not generate a lot of load.