Xen/IA64 upstream merge

Goal

Add paravirtualization support to Linux/IA64 with paravirt_ops and then integrate ia64/xen both domU/dom0 support.

Background

Linux/x86 supports paravirtualization technology as virtualization has become common. After some methods proposed, it finally adopted paravirt_ops solutions which Rusty Russel proposed uses a set of function pointer tables. For paravirt_ops/x86 see XenParavirtOps.

There are also some paravirtualization technologies available on on IA64 (IPF: Itanium Processor Family) architecture. e.g. Xen/IA64. Before there was a attempt to merge xen paravirtualization support to linux upstream. However things has changed because the x86 paravirtualization support has been included. So we want to push ia64 paravirtualization support following to the x86 paravirtualization support.

Strategy

At first minimize work/patchsize for domU merge. Then go further for optimized domU and dom0.

First Phase: domU merge
- minimize modification.
- postpone optimization where possible
Later Phases
1. domU optimization
2. dom0 support (x86 doesn't support yet, though.)

Repository

The following repo is available. That is based on git://git.kernel.org/pub/scm/linux/kernel/git/aegl/linux-2.6 test. And the current working branch is ia64-pv-ops-2008oct20-xen-ia64-optimized-domu which includes the changesets from the above test branch and git://git.kernel.org/pub/scm/linux/kernel/git/x86/linux-2.6-tip.git x86/xen branch.

http://people.valinux.co.jp/~yamahata/xen-ia64/for_eagl/linux-2.6-ia64-pv-ops.git/
- git clone http://people.valinux.co.jp/~yamahata/xen-ia64/for_eagl/linux-2.6-ia64-pv-ops.git/

There are also two repositories available. Those two are same. The current working branch is xen-ia64-domu-minimal-2008may09. They are old and the current development is done using the above repot.

http://gitorious.org/projects/linux-2-6-xen-ia64-pv-ops
- git clone git://gitorious.org/linux-2-6-xen-ia64-pv-ops/mainline.git
- git clone http://git.gitorious.org/linux-2-6-xen-ia64-pv-ops/mainline.git
http://people.valinux.co.jp/~yamahata/xen-ia64/linux-2.6-xen-ia64.git/ (It is discouraged to use this repo. This is only for those who have problems accessing the gitorious repo.)
- git clone http://people.valinux.co.jp/~yamahata/xen-ia64/linux-2.6-xen-ia64.git/
http://repo.or.cz/w/pv_ops_mirror.git (the mirror of the above.)
- git clone git://repo.or.cz/pv_ops_mirror.git
- git clone http://repo.or.cz/r/pv_ops_mirror.git

Related Mailing Lists

xen-ia64-devel
linux-ia64
Linux-foundation virtualization
kvm-ia64
Linux-kernel(LKML)

Xen/IA64 upstream merge To Do list

first phase task list: this phase was finished.

This phase was finished because all the patches were committed to the ia64 tree.

Task	Status	Developer	Description
Forward port from linux-2.6.18-xen	Done
common part	Done		the patches are now in xen maintainer queue waiting for x86.git merge.
pv_info	Done
pv_init_ops	Done
pv_cpu_asm_ops (ivt.S)	Done		freeing unused pv ivt.S is postponed to the next phase
pv_cpu_asm_ops (entry.S)	Done
pv_cpu_ops	Done		indirect C function call
pv_iosapic_ops	Done
xen irq chip(pv_irq_ops)	Done		should be renamed to other name. it conflicts x86 pv_irq_ops meaning.
pv_time_ops	Done		steal time accounting. share code with x86 if possible.
multi entry point	Done		allow non standard kernel entry point to boot as xen domU like xenLinux/x86
revise boot protocol	Postponed		postponed to the second phase. If required by the upstream, will address.
PAL/SAL	No need		found unnecessary
Documentation	Done		documentation on pv_ops/ia64 to help with maintainability
.S paravirtualization check tool	Done		a tool that can check to make sure people don't add direct uses of instructions to .S files that need to be paravirtulized

second phase task list: this phase was finished.

The planned patch except postponed ones have been merged. They will be available as 2.6.30, so this phase was finished.

Task	Status	Developer	Description
fsys.S	Done		paravirtualize fast systemcall path. fsys_fallback_syscall, fsys_bubble_down
gate page(gate.S)	Done		paravirtualize systemcall gate page.
binary patching for optimization	Done		define calling convention
save/restore	Done		includes live migration
balloon expansion	Postponed until dom0 or x86		the upstream balloon driver uses memory hot plug which breaks the xen/ia64 assumption. save/restore/dump-core will break.
ar.itc	Done		This is required for save/restore
sched_clock	Done		ar.itc is used to calculate sched_clock. But on xen environment, vcpu might be moving around on pcpus so that ar.itc isn't stable as expected. Should CONFIG_HAVE_UNSTAGBLED_SCHED_LOCK be enabled?
free unused page	Postponed until dom0		free memory which contains codes for unused pv instances. first candidates are ivt page and gate page. Hopefully free whole unused pv code.
revise boot protocol	Postponed until dom0		extend struct ia64_boot. postponed from the first phase
pal_call_static	Postponed until dom0
irq balance	Postponed until dom0		Currently xen irq chip doesn't support irq affinity so that irqbalance causes a bunch of kernel error messages and thus the system boot sequence doesn't complete. fix it.
pv spin lock	Postponed until dom0		paravirtualize spinlocks like x86

dom0 phase task list: current phase

Current phase. At first take a look at xen/x86 code.

Task	Status	Developer	Description
ioremap
pci device scanning
DMA			p2m, swiotlb, sba_iommu
kexec/kdump
PMU/PMD/PMC			xenoprof
static register calling convention
mca

How to use

Some hints for those who are brave to give it a try.

Preparing Xen environment

Set up hypervisor, dom0 and domU using the xen upstream one from source tree.
- For details, see XenIA64/HowTo/, XenOnia64, FedoraXenIA64 Probably the release(3.2?) tree might work.
Get the above git tree and build the domU kernel using the included config file(xen_domu_wip_defconfig).
Boot domU with the newly create kernel by the above step.

The recommended development environment

Here is my environment.

Debian etch on x86_64
with cross toolchains from embedian http://www.emdebian.org/index.html.

For cross compiling, see CrossCompiling. If you cross compile only Linux kernel tree you can by typing

make CROSS_COMPILE=ia64-linux-gnu- ARCH=ia64 <target>

Here you may want to change CROSS_COMPILE value depending on your environment.

Add the entry to your apt-line and install cross toolchains by apt-get. For details see http://www.emdebian.org/tools/crosstools.html . I'm using following version

binutils-ia64-linux-gnu
cpp-4.1-ia64-linux-gnu
g++-4.1-ia64-linux-gnu
gcc-4.1-ia64-linux-gnu
gcc-4.1-ia64-linux-gnu-base