Xen Cross-Compiling HOWTO

This HOWTO explains installing and using a cross-compiler for Xen. To build the hypervisor and kernel, only a compiler and linker are needed. To build the tools, a full sys-root environment is needed, including the header files and libraries needed for the tools compilation.

The examples in this document assume a target architecture of ia64, building in a host environment of Fedora8/x86_64 or Etch/x86_64. For the sake of simplicity and avoiding license issues, the sys-root uses Debian etch in either case, but it's not difficult to use another distribution, provided it's available for your target arch.

Install a sys-root

It's possible to build the hypervisor and kernel without the sys-root, but a full cross-build of Xen including the tools will need the sys-root for userland includes and libraries.

Pre-built Debian/ia64 sys-root

You can download a prebuilt Debian/ia64 (etch) root filesystem per the steps below.

wget http://free.linux.hp.com/~agriffis/cross/sys-roots/etch-ia64.tar.gz
mkdir /usr/ia64-linux-gnu
tar xzf etch-ia64.tar.gz -C /usr/ia64-linux-gnu
ln -s etch-ia64 /usr/ia64-linux-gnu/sys-root

Note that this root filesystem isn't labelled correctly for selinux, so it won't work on a system with selinux enabled. To disable selinux temporarily, run "setenforce 0". To disable it permanently, change /etc/selinux/config to read "SELINUX=permissive".

Build your own sys-root

There are many ways to skin this cat. One possibility is to use repack_cross.pl from Lennert's email (see the fedora-devel-list archive link above). This would take a lot of effort to gradually build up the sys-root to contain all the dependencies required by the xen-tools build. In the process, you'd gradually fill in the missing dependencies from the cross-compiler installation.

A much easier approach is to install an ia64 system then copy the root filesystem to /usr/ia64-linux-gnu/sys-root. The Debian/ia64 sys-root above was made using the following steps on a Debian ia64 system. Note you shouldn't need to do this, since it was done for you...

apt-get install debootstrap
mkdir /chroots/etch-ia64
debootstrap --verbose --include=build-essential /chroots/etch-ia64
chroot /chroots/etch-ia64
apt-get install build-essential
apt-get build-dep xen-tools
apt-get build-dep linux-image-2.6-mckinley
apt-get install libncurses5-dev libssl-dev libx11-dev pciutils-dev libgnutls-dev

# Fix symlinks in /usr/lib to be relative (../../lib/*) instead of absolute
# (/lib/*).  Otherwise the cross-linker will complain about incompatible objects
# when this filesystem is mounted on /usr/ia64-linux-gnu/sys-root
apt-get install symlinks
symlinks -rc /usr/lib

Another valid way to make the sys-root is to install a full RHEL5.1 system and tar up the root filesystem. You'll need to do the symlinks step above in order to fix the absolute symlinks in /usr/lib.

In my experiments, Fedora8/ia64 doesn't work as a sys-root because libunwind is missing some bits, and CentOS4.6/ia64 doesn't work because some of its libraries are too old. YMMV.

Install a cross-toolchain on Fedora8

Pre-built toolchain rpms

Cross compilers for Fedora8/x86_64 (and for i386 if that suits you) are available at http://free.linux.hp.com/~agriffis/cross/. These are based on previous work by David Woodhouse and Lennert Buytenhek, as posted at http://www.redhat.com/archives/fedora-devel-list/2007-October/msg00045.html (and integrated back into their sources as of https://www.redhat.com/archives/fedora-devel-list/2008-April/msg00334.html).

As root, do:

cd /etc/yum.repos.d
wget http://free.linux.hp.com/~agriffis/cross/agriffis-cross.repo

Normally the next step would be yum install ia64-linux-gnu-gcc. However this command will yield an error message because the gcc rpm claims some dependencies for building userland programs; in particular, it wants glibc in the sys-root. Since you don't need that for building the hypervisor and/or kernel, you can get around those deps by using these commands:

yum install yum-utils
mkdir /root/agriffis-cross
cd /root/agriffis-cross
yumdownloader --resolve --tolerant ia64-linux-gnu-gcc
rpm -Uvh --nodeps *.rpm

yumdownloader is able to fetch all the dependencies provided in the yum repository, while harmlessly ignoring the dependencies that won't be met. (This doesn't appear to be possible with yum directly.) Then rpm --nodeps can install the downloaded rpms. After they're installed, you can remove the agriffis-cross directory.

Install a cross-toolchain on Debian Etch

The [Emdebian project] provides a range of cross-toolchains. Unfortunately the Emdebian toolchain isn't built for a sys-root, so it suffices for hypervisor and kernel, but to build the tools you need to rebuild the toolchain to use the sys-root.

First, install the emdebian toolchain since this will provide some deps:

echo 'deb http://www.emdebian.org/debian/ stable main' >> /etc/apt/sources.list
wget -O- http://www.emdebian.org/0x97BB3B58.txt | apt-key add -
aptitude update
aptitude install gcc-4.1-ia64-linux-gnu

Second, rebuild binutils with sys-root support (see "Install a sys-root" above):

echo 'deb-src ftp://ftp.us.debian.org/debian/ stable main' >> /etc/apt/sources.list
aptitude update
apt-get build-dep binutils
apt-get source binutils
cd binutils*/
sed -i '/\$(ADDITIONAL_TARGETS)/s/\$/--with-sysroot=yes /' debian/rules
TARGET=ia64 fakeroot debian/rules binary-cross  # see debian/README.cross
cd ..
dpkg -i binutils-ia64-linux-gnu*deb

Third, rebuild gcc with sys-root support:

aptitude install dpkg-cross
wget ftp://ftp.us.debian.org/debian/pool/main/liba/libatomic-ops/libatomic-ops-dev_1.1-4_ia64.deb
dpkg-cross -i -a ia64 libatomic-ops-dev_1.1-4_ia64.deb
apt-get build-dep gcc-4.1
apt-get source gcc-4.1
cd gcc-4.1*/
export GCC_TARGET=ia64 DEB_CROSS_INDEPENDENT=yes with_sysroot=yes
debian/rules  # see debian/README.cross
dpkg-buildpackage
cd ..
dpkg -i *amd64.deb *all.deb

Cross-build Xen

Build the hypervisor and kernel

With the cross-compiler installed, building the hypervisor and kernel is simple. It's the same as a normal build except that XEN_TARGET_ARCH and CROSS_COMPILE are specified, as follows:

make XEN_TARGET_ARCH=ia64 CROSS_COMPILE=ia64-linux-gnu- xen
make XEN_TARGET_ARCH=ia64 CROSS_COMPILE=ia64-linux-gnu- kbuild

Note that the ia64 kernel build reports a couple of inconsequential errors. These are the result of a bug which was recently fixed upstream, but hasn't been backported to linux-2.6.18-xen.hg. See http://article.gmane.org/gmane.linux.ports.ia64/17535

Build the tools

With the sys-root in place, building the tools is easy too:

make XEN_TARGET_ARCH=ia64 CROSS_COMPILE=ia64-linux-gnu- tools

or build the entire thing with:

make XEN_TARGET_ARCH=ia64 CROSS_COMPILE=ia64-linux-gnu- world