The installation of Xen 64 bit on Debian Etch (4.0)
Mostly taken from the excellent tutorial at "The perfect xen 3.1.0 setup”
Although 3.1.0 is an older implementation, there was a large difference in download size and the new package is quite different enough for me to install the older one.
My comments are in italics.
One major difference with my installation is that I installed the 64 bit version.
The 64 bit version will allow you up to 64GB RAM using the Intel Core 2 duo chip, and I believe it's even more if you choose the AMD 64 bit CPUs, as the latter are true 64 bit, whereas Intel's CPUs apparently only have 48 bit memory space. If you have the 32 bit there is a limitation on the amount of memory able to be accessed by the system. If you have 32 bit linux with pae extensions enabled (it's a kernel module for accessing more memory than 3.6GB) you can get around this 32 bit limitation, or you can just install a 64 bit OS which is able to see a heap of RAM as standard.
All packages you'll be installing for the 64 bit version on Debian will end in "amd64", not "i386".
The Perfect Xen 3.1.0 Setup For Debian Etch (i386)
Xen lets you create guest operating systems (*nix operating systems like Linux and FreeBSD), so called "virtual machines" or domUs, under a host operating system (dom0). Using Xen you can separate your applications into different virtual machines that are totally independent from each other (e.g. a virtual machine for a mail server, a virtual machine for a high-traffic web site, another virtual machine that serves your customers' web sites, a virtual machine for DNS, etc.), but still use the same hardware. This saves money, and what is even more important, it's more secure. If the virtual machine of your DNS server gets hacked, it has no effect on your other virtual machines. Plus, you can move virtual machines from one Xen server to the next one.
I will use Debian Etch (i386) for both the host OS (dom0) and the guest OS (domU). I will describe how to install Xen from the binary package in chapter 5 and from the sources (chapter 5). In an additional section at the end of chapters 4 and 5 I will also show how to create a virtual local network with virtual machines, with dom0 being the router. Chapters 4 and 5 are mutually exclusive. [ note: very, very easy to install the packages. These are version 3.0.3 and install without a hitch]
This howto is meant as a practical guide; it does not cover the theoretical backgrounds. They are treated in a lot of other documents in the web. This document comes without warranty of any kind! I want to say that this is not the only way of setting up such a system. There are many ways of achieving this goal but this is the way I take. I do not issue any guarantee that this will work for you!
Install The Debian Etch Host System (dom0)
You can overall follow these instructions, but with a few changes:
http://www.howtoforge.com/perfect_setup_debian_etch
http://www.howtoforge.com/perfect_setup_debian_etch_p2
dom0's FQDN in this example will be server1.example.com, so I specify server1 as Hostname and example.com as Domain name. server1.example.com's IP address will be 192.168.0.100 in this tutorial.
When it comes to the partitioning, I select Manual. I create the following partitions:
/boot 150 MB (Primary) (Location for the new partition: Beginning) (ext3) (Bootable flag: on <-- important, otherwise your system will not boot!) note: must be ext2/3 or grub won't boot
swap 1GB (Logical) (Location for the new partition: Beginning)
/ 3GB (Logical) (Location for the new partition: Beginning) (ext3 reiserfs, noatime)
/vserver the rest (Logical) (Location for the new partition: Beginning) (ext3 xfs,noatime)
When the Debian installer prompts Choose software to install: I make no selection and go on (dom0 should run as few software as possible in order not to be vulnerable to attacks. To the outside world it will be accessible only over SSH.).
Preparation Of The Host System
After the base system has been installed and you have removed the installation CD and booted the base system, you should run
apt-get install ssh openssh-server vim-full to install an SSH server so that you can connect to the system with an SSH client such as PuTTY. The vim-full package makes sure that the vi text editor behaves as expected (without vim-full, you might experience some strange behaviour in the vi text editor).
Because the Debian Etch installer has configured our system to get its network settings via DHCP, we have to change that now because a server should have a static IP address. Edit /etc/network/interfaces and adjust it to your needs (in this example setup I will use the IP address 192.168.0.100):
vi /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
# The loopback network interface
auto lo
iface lo inet loopback
# The primary network interface
#allow-hotplug eth0
#iface eth0 inet dhcp
auto eth0
iface eth0 inet static
- address 192.168.0.100 netmask 255.255.255.0 network 192.168.0.0 broadcast 192.168.0.255 gateway 192.168.0.1
Then restart your network:
/etc/init.d/networking restart
Then edit /etc/hosts. Make it look like this:
vi /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.0.100 server1.example.com server1
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
Edit /etc/resolv.conf and add some nameservers:
vi /etc/resolv.conf
nameserver 213.191.92.84
nameserver 145.253.2.75
nameserver 193.174.32.18
Now run
echo server1.example.com > /etc/hostname
and reboot the system:
shutdown -r now
Afterwards, run
hostname
hostname -f
Both should show server1.example.com.
From now on you can use an SSH client such as PuTTY and connect from your workstation to your Debian Etch server and follow the remaining steps from this tutorial.
Edit /etc/apt/sources.list. Comment out the CD. It should look like this:
vi /etc/apt/sources.list
#
# deb cdrom:[Debian GNU/Linux 4.0 r0 _Etch_ - Official i386 NETINST Binary-1 20070407-11:29]/ etch contrib main
#deb cdrom:[Debian GNU/Linux 4.0 r0 _Etch_ - Official i386 NETINST Binary-1 20070407-11:29]/ etch contrib main
deb http://ftp.monash.edu.au/pub/linux/debian/ etch main contrib non-free
# extremely fast at Melbourne Universities!
deb http://security.debian.org/ etch/updates main contrib
deb-src http://security.debian.org/ etch/updates main contrib
Then run
apt-get update
to update the apt package database and
apt-get upgrade
to install the latest updates (if there are any).
Install Xen
There are two ways to install Xen: install the binary package from the Xen website, or compile Xen and the Xen kernels from the sources.
The first way is easier. In my past Xen tutorials I always recommended to compile Xen from the sources because some features such as quota or iptables were missing in the binary package. This has changed now, so you will most likely find all needed features in the binary package, so I recommend to install the binary package. If you want to be in full control of what features are in the Xen kernel or if you need a special feature that's not in the binary package, you should compile Xen from the sources. In chapter 4 I describe how to install the Xen binary package which might be easier for beginners. In chapter 5 I describe how to compile and install Xen from the sources. Chapters 4 and 5 are mutually exclusive. Follow either chapter 4 or 5, but not both of them! note: binary distribution (chapter 4) is missing some really important sata modules and entirely failed to boot on the test system
About Xen-tools
xen-tools is a collection of simple perl scripts which allow you to easily create new guest Xen domains upon your Debian GNU/Linux host.
Once installed and configured you can create a new Xen instance in a matter of minutes. Each new Xen domain will be complete with:
All networking details setup, with either multiple static IP addresses or DHCP.
An installation of OpenSSH.
An arbitary set of partitions.
Your new instance will be completed by having the user accounts from your guest system copied over, and you may optionally boot the image as soon as it has been created.
Installation Methods
Mutiple installation methods are supported to increase your choices. You can choose to:
Install via debootstrap
Install via rpmstrap
Copy a previously created image.
Untar an archive of a pristine image.
note: rpmstrap will not work for modern rpm-based distros. I Created the image instead with qemu.
Flexibility
Because the creation of new instance is very modular there are very few dependencies, and adding new flavours of GNU/Linux to install should be very simple.
Please read the FAQ for more details
Contents
1.Introduction
2.Dom0 point-to-point to each DomU
3.DomU and Dom0 on virtual network
4.DomU and Dom0 on real network
5.Point-to-point between two DomU
6.Virtual Network between DomU
7.Everything
Introduction
I try to put on this pages informations about configuration for differents network targets. This is for Xen 3 unless specified.
I think most of the time creating vethX//vifX.0 is useless, they should be created on demand.
For a DomU named peter, I would prefer an interface named vif-peter.0 instead of vifX.0
Dom0 point-to-point to each DomU
The goal is to have a single link between one Dom0 and one DomU
DomU and Dom0 on virtual network
The goal is to have many DomU and Dom0 on the same virtual network. Dom0 is acting as a router/firewall.
There is a solution shown here: http://en.opensuse.org/Xen3_and_a_Virtual_Network
DomU and Dom0 on real network
The goal is to have DomU on the same real network as Dom0 and other real computers.
Point-to-point between two DomU
The goal is to have a link between two DomU. Dom0 is not supposed to RX, TX or do filter on this network.
Virtual Network between DomU
The goal is to create a virtual network between many DomU. Dom0 is not supposed to RX, TX or do filter on this network.
Everything
The goal is to have a mix of differents links.
How to Build Version 3.2 from source.
Pre-reqs, from http://fresh.t-systems-sfr.com/unix/misc/xen-3.2.0.tar.gz:a/xen-3.2.0/README:
Quick-Start Guide - Source Release
==================================
First, there are a number of prerequisites for building a Xen source release. Make sure you have all the following installed, either by visiting the project webpage or installing a pre-built package provided by your Linux distributor:
- GCC v3.4 or later
- GNU Make
- GNU Binutils
- Development install of zlib (e.g., zlib-dev)
- Development install of Python v2.3 or later (e.g., python-dev)
- Development install of curses (e.g., libncurses-dev)
- Development install of openssl (e.g., openssl-dev)
- Development install of x11 (e.g. xorg-x11-dev)
- bridge-utils package (/sbin/brctl)
- iproute package (/sbin/ip)
- hotplug or udev
The perfect install says:
apt-get remove exim4 exim4-base lpr nfs-common portmap pidentd pcmcia-cs pppoe pppoeconf ppp pppconfig
apt-get install iproute bridge-utils python-twisted binutils zlib1g-dev python-dev transfig bzip2 screen ssh debootstrap libcurl3-dev libncurses5-dev x-dev build-essential gettext gawk pciutils-dev
The added installation of gawk is necessitated by some instruction or other that's missing in the regular version when you try make prep-kernels following.
From here ==== So I devised with the following process to build Xen 3.2.
% mkdir build
% cd build
% wget http://bits.xensource.com/oss-xen/release/3.2.0/xen-3.2.0.tar.gz
% tar xzf xen-3.2.0.tar.gz
% # Download the xenifid kernel tree manually, but NOT in xen-3.2.0/
% # because the buildconfig/select-repository script would skip it.
% # ! xen-3.2.0/ and linux-2.6.18-xen.hg/ must be at the same level !
% hg clone http://xenbits.xensource.com/linux-2.6.18-xen.hg
% cd xen-3.2.0
% make prep-kernels
% cp /boot/config-2.6.18-my build-linux-2.6.18-xen_x86_32/.config
% make dist
That last bit takes a really long time, mostly compiling kernel modules. It failed the first time due to the x11-dev not being installed. Failed again because pciutils-dev wasn't installed. Basically, if you don't install the packages above, it will definitely waste your time and efforts.
Make an initrd based on the new version of the kernel (look in /boot for this).
depmod 2.6.18.8-xen
mkinitrd.yaird -o /boot/initrd.img-2.6.18.8-xen 2.6.18.8-xen
update-grub # check that /boot/grub/menu.lst is OK here. It may well add undesireable entries. Mine did!
Install using packages on x86_64.
Date 22/2/2008
Problem #1: Installer wouldn't work. Stopped with a pci error.
Solution: Install with “expert all-generic-ide pci=nommconf”
Note: This is a very common x64 error seen on a lot of new motherboards I have encountered.
Booted net install and used monash as the mirror. Very, very fast (> 10Mbit transfer), must use proxy set, if applicable. The URL is above (apt/sources.list). If you must use a proxy, but don't know how to do it from the command line, the quick way is to issue export http_proxy="http://user:password@proxy:port" ; export ftp_proxy="http://user:password@proxy:port". The permanent way is to add it your .profile then do a ". .profile" to activate the changes.
Installed xen-linux-system-2.6.18-4-xen-amd64, xen-tools
Rebooted and select the xen kernel (it won't do this automatically – update it with grub-set-default).
For xen-3.1 do the following:
Installing The Binary Package ''Reminder: binary kernel from xensource doesn't work with sata disks''
Run the following commands:
apt-get remove exim4 exim4-base lpr nfs-common portmap pidentd pcmcia-cs pppoe pppoeconf ppp pppconfig
apt-get install screen ssh debootstrap python python-twisted iproute bridge-utils libcurl3-dev libssl0.9.7
and the really important one:
apt-get install xen-linux-system-2.6.18-6 xen-tools xen-utils
Xen is now installed. Check grub and make sure the menu.lst is correct. I found that sometimes grub-update mixes the kernel entries up a bit. Just to be sure.
Now reboot the system:
reboot
At the boot prompt, Grub should now list Xen 3.1.0 / Debian GNU/Linux, kernel 2.6.18-xen as the first kernel and boot it automatically:
If your system comes up without problems, then everything is fine!
Run uname -a
and your new Xen kernel should show up:
server1:~# uname -a
Linux server1.example.com 2.6.18-xen #1 SMP Fri May 18 16:11:33 BST 2007 amd64 GNU/Linux
Creating Virtual Machines (domU)
We will use xen-tools to create virtual machines. xen-tools make it very easy to create virtual machines - please read this tutorial to learn more:
http://www.howtoforge.com/xen_tools_xen_shell_argo.
Next we edit /etc/xen-tools/xen-tools.conf. This file contains the default values that are used by the xen-create-image script unless you specify other values on the command line. I changed the following values and left the rest untouched:
vi /etc/xen-tools/xen-tools.conf
[...]
gateway = 192.168.0.1
netmask = 255.255.255.0
passwd = 1
kernel = /boot/vmlinuz-2.6.18-xen
#initrd = /boot/initrd.img-2.6.16-2-xen-686
mirror = ftp://ftp.monash.edu.au/pub/debian/
Although the original text says "Please make sure that you comment out the initrd line!" All of my test installations needed a ramdisk for virtual machines.
The passwd = 1 line makes that you can specify a root password when you create a new guest domain. In the kernel line you must specify the domU kernel that you want to use for your guest domains. In the mirror line specify a Debian mirror close to you.
Make sure you specify a gateway and netmask. If you don't, and you don't specify a gateway and netmask on the command line when using xen-create-image, your guest domains won't have networking even if you specified an IP address!
Now let's create our first guest domain, xen1.example.com, with the IP address 192.168.0.101:
xen-create-image --hostname=xen1.rmit.edu.au --size=2Gb--swap=256Mb --ide --ip=192.168.0.101 --netmask=255.255.255.0 --gateway=192.168.0.1 --force
--dir=/vserver --memory=128Mb --arch=amd64 --kernel=/boot/vmlinuz-2.6.18-6-xen-vserver-amd64 --initrd=/boot/initrd.img-2.6.18-6-xen-vserver-amd64 --debootstrap
--dist=etch --mirror=ftp://ftp.monash.edu.au/pub/linux/debian --passwd
A lot of switches are unnecessary here because we specified the same details in /etc/xen-tools/xen-tools.conf but it shows that you can specify the desired settings either on the command line or in /etc/xen-tools/xen-tools.conf. Please make sure that you specify --ide, otherwise your virtual machine might not boot!
(To learn more about the available options, take a look at the xen-create-image man page: man xen-create-image)
The xen-create-image command will now create the xen1.example.com virtual machine for us. This can take a few minutes. The output should be similar to this one:
server1:~# xen-create-image --hostname=xen1.example.com --size=2Gb --swap=256Mb --ide \
> --ip=192.168.0.101 --netmask=255.255.255.0 --gateway=192.168.0.1 --force \
> --dir=/vserver --memory=32Mb --arch=i386 --kernel=/boot/vmlinuz-2.6.18-xen \
> --debootstrap --dist=etch --mirror=http://ftp2.de.debian.org/debian/ --passwd
General Infomation
Hostname : xen1.example.com
Distribution : etch
Fileystem Type : ext3
Size Information
Image size : 2Gb
Swap size : 256Mb
Image type : sparse
Memory size : 32Mb
Kernel path : /boot/vmlinuz-2.6.18-xen
Networking Information
IP Address 1 : 192.168.0.101
Netmask : 255.255.255.0
Gateway : 192.168.0.1
WARNING
Loopback module not loaded and you're using loopback images
Run the following to load the module:
modprobe loop loop_max=255
Creating swap image: /vserver/domains/xen1.example.com/swap.img
Done
Creating disk image: /vserver/domains/xen1.example.com/disk.img
Done
Creating ext3 filesystem on /vserver/domains/xen1.example.com/disk.img
Done
Installing your system with debootstrap mirror ftp://ftp.monash.edu.au/debian/
Done
Running hooks
Done
No role script specified. Skipping
Creating Xen configuration file
Done
Setting up root password
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
All done
Logfile produced at:
- /var/log/xen-tools/xen1.example.com.log
server1:~#
(You can ignore this warning:WARNING
Loopback module not loaded and you're using loopback images
Run the following to load the module:
modprobe loop loop_max=255
The virtual machine will work nevertheless.)
There should now be a xen1.example.com configuration file - /etc/xen/xen1.example.com.cfg. Take a look at it to become familiar with virtual machines
configuration files:
cat /etc/xen/xen1.example.com.cfg
#
# Configuration file for the Xen instance xen1.example.com, created on
# Mon May 28 20:53:05 2007.
#
#
# Kernel + memory size
#
kernel = '/boot/vmlinuz-2.6.18-xen'
memory = '32'
#
# Disk device(s).
#
root = '/dev/hda1 ro'
disk = [ 'file:/vserver/domains/xen1.example.com/disk.img,hda1,w', 'file:/vserver/domains/xen1.example.com/swap.img,hda2,w' ]
#
# Hostname
#
name = 'xen1.example.com'
#
# Networking
#
vif = [ 'ip=192.168.0.101' ]
#
# Behaviour
#
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
To start the virtual machine, run
xm create /etc/xen/xen1.example.com.cfg
Run
xm console xen1.example.com
to log in on that virtual machine (type CTRL+] if you are at the console, or CTRL+5 if you're using PuTTY to go back to dom0), or use an SSH client to connect to it (192.168.0.101).
To get a list of running virtual machines, type
xm list
The output should look like this:
server1:~# xm list
Name ID Mem VCPUs State Time(s)
Domain-0 0 301 1 r
- 1191.0
xen1.example.com 1 32 1 -b
- 50.6
server1:~#
To shut down xen1.example.com, do this:
xm shutdown xen1.example.com
If you want vm01 to start automatically at the next boot of the system, then do this:
ln -s /etc/xen/xen1.example.com.cfg /etc/xen/auto
Here are the most important Xen commands:
xm create -c /path/to/config - Start a virtual machine.
xm shutdown <name> - Stop a virtual machine.
xm destroy <name> - Stop a virtual machine immediately without shutting it down. It's as if you switch off the power button.
xm list - List all running systems.
xm console <name> - Log in on a virtual machine.
xm help - List of all commands.
Let's create a second vm, xen2.example.com with the IP address 192.168.0.102:
xen-create-image --hostname=xen2.example.com --size=2Gb --swap=256Mb --ide \
--ip=192.168.0.102 --netmask=255.255.255.0 --gateway=192.168.0.1 --force \
--dir=/vserver --memory=32Mb --arch=i386 --kernel=/boot/vmlinuz-2.6.18-xen \
--debootstrap --dist=etch --mirror=http://ftp2.de.debian.org/debian/ --passwd
Afterwards, you can start xen2.example.com like this:
xm create /etc/xen/xen2.example.com.cfg
and shut it down like this:
xm shutdown xen2.example.com
A list of all virtual machines that were created with the xen-create-image command is available under
xen-list-images
server1:~# xen-list-images
Name: xen1.example.com
Memory: 32
IP: 192.168.0.101
Name: xen2.example.com
Memory: 32
IP: 192.168.0.102
server1:~#
Problems
1. The ethernet card I have works perfectly in the xen 32 bit version and the non-xen 64 bit version. It does not and has not ever worked with the Xen 64 bit version. Blowed if I know why. I even downloaded an rpm of the known working version from Fedora and it still failed on Debian. Same two exact kernels, same modules, copied directly and with Debian they fail. Who knows why? Apparently it's only this card and only the 64 bit Debian.
So, watch out for the Intel 82566DM gigabit ethernet chip.
2. USB keyboard incompatible. It slowed the response time for both KB and mouse down by about 99%. System was unuseable until I replaced the USB KB with a good old-fashioned PS2 version. Again, only the Debian 64 bit version.
I gave these problems a lot of time. Re-compiled kernels, installed new versions of Xen (both 3.1 and 3.2). Nothing ever made any difference to either problem. Even tried disabling USB to see if the two things were an IRQ sharing problem and thus related. No difference. If you have a board with built in 82566DM and you use a USB keyboard, I suggest you get a PS/2 adaptor (or a new PS/2 KB) and a new ethernet card, disabling the built-in one. If your server is to have lots of bandwidth and VMs, you should look at getting a multi port add-on card.
