Xen Development Projects

• Task 1: Upstream PVSCSI scsifront frontend driver (for domU).
• Task 2: Upstream PVSCSI scsiback backend driver (for dom0).
• Send to various related upstream mailinglists for review, comments.
• Fix any upcoming issues.
• Repeat until merged to upstream Linux kernel git tree.
• http://git.kernel.org/?p=linux/kernel/git/konrad/xen.git;a=shortlog;h=refs/heads/devel/xen-scsi.v1.0
• More info: http://wiki.xen.org/xenwiki/XenPVSCSI

• Upstream PVUSB usbfront frontend driver (for domU).
• Upstream PVUSB usbback backend driver (for dom0).
• Send to various related upstream mailinglists for review, comments.
• Fix any upcoming issues.
• Repeat until merged to upstream Linux kernel git tree.
• http://git.kernel.org/?p=linux/kernel/git/konrad/xen.git;a=shortlog;h=refs/heads/devel/xen-usb.v1.1
• More info: http://wiki.xen.org/xenwiki/XenUSBPassthrough

• Task 1: Implement PVSCSI in xl/libxl, make it functionally equivalent to xm/xend.
• Send to xen-devel mailinglist for review, comments.
• Fix any upcoming issues.
• Repeat until merged to xen-unstable.
• See above for PVSCSI drivers for dom0/domU.
• Xen PVSCSI supports both PV domUs and HVM guests with PV drivers.
• More info: http://wiki.xen.org/xenwiki/XenPVSCSI

• Task 1: Implement PVUSB in xl/libxl, make it functionally equivalent to xm/xend.
• Send to xen-devel mailinglist for review, comments.
• Fix any upcoming issues.
• Repeat until merged to xen-unstable.
• See above for PVUSB drivers for dom0/domU.
• Xen PVUSB supports both PV domUs and HVM guests with PV drivers.
• More info: http://wiki.xen.org/xenwiki/XenUSBPassthrough

• Multiple disks in a guest cause contention in the global pool of pages.
• There is only one ring page and with SSDs nowadays we should make this larger, implementing some multi-page support.
• With multi-page it becomes apparent that the segment size ends up wasting a bit of space on the ring. BSD folks fixed that by negotiating a new parameter to utilize the full size of the ring. Intel had an idea for descriptor page.
• Add DIF/DIX support [1] for T10 PI (Protection Information), to support data integrity fields and checksums.
• Further perf evaluation needs to be done to see how it behaves under high load.
• Further discussion and issues outlined in http://lists.xen.org/archives/html/xen-devel/2012-12/msg01346.html

• http://lists.xen.org/archives/html/xen-devel/2012-01/msg02561.html
• http://www.spinics.net/lists/linux-nfs/msg22575.html

Multiqueue support allows a single virtual network interface (vif) to scale to multiple vcpus. Each queue has it's own interrupt, and thus can be bind to a different vcpu. KVM VirtIO, VMware VMXNet3, tun/tap and various other drivers already support multiqueue in upstream Linux.

Some general info about multiqueue: http://lists.linuxfoundation.org/pipermail/virtualization/2011-August/018247.html In the current implementation of Xen PV network, every vif is equipped with only one TX/RX ring pair and one event channel, which does not scale when a guest has multiple vcpus. If we need to utilize all vcpus to do network job then we need to configure multiple vifs and bind interrupts to vcpus manuals. This is not ideal and involves too much configuration. The multiqueue support in Xen vif should be straight forward. It requires changing the current vif protocol and the code used to initialize / connect / reconnect vifs. However, there are risks in terms of collaboration, it is possible multiple parties will work on same piece of code. Here are possible obstacles and thoughts:

• netback worker model change - the possible change is from M:N to 1:1 is not really an obstacle because 1:1 is just a special case for M:N
• netback page allocation mechanism change - not likely to have protocol change
• netback zero-copy - not likely to have protocol change
• receiver-side copy - touches both protocol and implementation,
• multi-page ring - touches protocol and implementation, should be easy to merge
• split event channel - touches protocol and implementation, should be easy to merge

• have multi-queue patch ready to upstream or upstreamed
• benchmark report (basic: compare single-queue / multi-queue vif. advanced: compare Xen multi-queue vif against KVM multi-queue VirtIO etc.)

• Multiple disks in a guest cause contention in the global pool of pages.
• There is only one ring page and with SSDs nowadays we should make this larger, implementing some multi-page support.
• With multi-page it becomes apparent that the segment size ends up wasting a bit of space on the ring. BSD folks fixed that by negotiating a new parameter to utilize the full size of the ring. Intel had an idea for descriptor page.
• Add DIF/DIX support [2] for T10 PI (Protection Information), to support data integrity fields and checksums.
• Further perf evaluation needs to be done to see how it behaves under high load.
• Further discussion and issues outlined in http://lists.xen.org/archives/html/xen-devel/2012-12/msg01346.html

Adding this component in the network or block backends can mean that we can keep the data longer in the cache

• Have upstream patches.
• benchmark report of with and without.

• Upstream patches
• perf --xen top working.

For more informations please check:

• Linux git revision 8eaffa67b43e99ae581622c5133e20b0f48bcef1
• http://lists.xen.org/archives/html/xen-devel/2012-06/msg01950.html

The goal is to create in the Linux kernel a general PAT lookup system where it can determine which page bits

A debug effort / end-goal would be to properly round-up dom0's vcpus, and be able to use the kgdb functionality to debug

Original reference thread:

power usage limit. This is particularly useful for data centers, where there may be a need to reduce power consumption based on availability of electricity or cooling. A more complete writeup is available at LWN.

These same arguments apply to Xen. The purpose of this project would be to implement a similar functionality in Xen, and to make it interface as well as possible with the Linux PowerClamp tools, so that the same tools could be used

Moreover, Xen offers with an implementation of one of the most famous and efficient real-time scheduling algorithm, the Earliest Deadline First (which is called SEDF in Xen), and real-time support is a key feature for a successful embedded hypervisor. Using such an advanced scheduling policy is, if it is implemented correctly, a great advancement and provide much more flexibility than only using vCPU pinning (which is what most embedded hypervisors do to guarantee real-time performances and isolation).

In the context of embedded virtualization, providing low latency (when reacting to external events) to the hosted VMs, and a good level of temporal isolation among the different hosted VMs is something fundamental. This means not only the failures in any VM should not affect any other one, but also that, independently of what a VM is doing, it shouldn't be possible for it to cause any other VM to suffer from higher (and potentially dangerous!) latency and/or impairing the capability of any other VM to respect its own real-time constraints.

Moreover, Xen offers with an implementation of one of the most famous and efficient real-time scheduling algorithm, the Earliest Deadline First (which is called SEDF in Xen), and real-time support is a key feature for a successful embedded hypervisor. Using such an advanced scheduling policy is, if it is implemented correctly, a great advancement and provide much more flexibility than only using vCPU pinning (which is what most embedded hypervisors do to guarantee real-time performances and isolation).

However, SEDF, the EDF implementation in Xen, is there, suffers from some rough edges. In fact, as of now, SEDF deals with events such as a vCPU blocking --in general, stopping running-- and unblocking --in general, restarting running-- by trying (and failing!) to special case all the possible situations, resulting in the code being rather complicated, ugly, inefficient and hard to maintain. Unified approaches have been proposed for enabling blocking and unblocking in EDF, while still guaranteeing temporal isolation among different vCPUs. It also lacks pproper multiprocessor support, meaning that it does not properly handle SMP systems, unless vCPU are specifically and statically pinned by the user. This is a big limitation of the current implementation, especially since EDF can work well without the need of imposing this constraint, providing much more flexibility and efficiency in exploiting the system resources to their most.

Ideally, each VM should have its memory allocated out of just one node and, as long as its vCPUs also run there, both throughput and latency are optimal. However, in cases where a VM ends up having its memory allocated from multiple nodes, we should inform it that it's running on a NUMA platform: a virtual NUMA. This could be very important, especially for some specific workloads (for instance, HPC applications). In fact, if the guest OS and application have any NUMA support, exporting the virtual topology is the only way to render that effective, and perhaps filling, at least to some extent, the gap in the performances introduced by the needs of distributing the guests on more than one node. Just for reference, this feature, under the name of vNUMA, is one of the key and most advertised ones of VMWare vSphere 5 (vNUMA: what it is and why it matters).

This project aims at introducing virtual NUMA in Xen. This has some non-trivial interaction with other aspects of the NUMA support of Xen itself, namely automatic placement at VM creation time, dynamic memory migration among nodes, and others, meaning that some design decision needs to be made. After that, virtual topology exposure will have to be implemented for all the kind of guests supported by Xen.

When it comes to memory, Xen offers a set of different mechanisms for over-committing the host memory, the most common, widely known and utililsed is ballooning. This has non-trivial interference with NUMA friendliness. For instance, when freeing some memory, current ballooning implementations try to balloon down existing guests, but that happens without any knowledge or consideration of on which node(s) the freed memory will end up being. As a result, we may be able to create the new domain, but not quite as able to place all its memory on a single node, as ballooning could well have freed half of the space on a node, and half on another. What we should instead do is try to make space "node-wise", i.e., ballooning down the VMs that would allow the new guest to fit into a node. Basically, we need to teach the whole ballooning mechanism about NUMA.

If possible/interesting, also the other technologies for memory overcommitment could be touched during this project. Page sharing, for instance. In fact, sharing pages between guests residing on different nodes is, in general, a bad idea, but there is nothing preventing for this to happen right now, and so that is what a candidate should design and implement.

If a workload is made up of more than just a VM, running on the same NUMA host, it might be best to have two (or more) VMs share a node, as well as right the opposite, depending on the specific characteristics of he workload itself, and this might be considered during placement, memory migration and perhaps scheduling.

The idea is that sometimes you have a bunch of VMs that would like to cooperate, and sometimes you have a bunch of VMs that would like to be kept as apart as possible from other VMs (competitive). In the cooperative VMs scenario, one wants to optimize for data flowing across VMs in the same host, e.g., because a lot of data copying is involved (a WebApp and DB VMs working together). This means trying to have VMs sharing data in the same node and, if possible, even in the same PCPU's caches, in order to maximize the memory throughput between the VMs. On the other hand, in the competitive VMs scenario, one wants to optimize for data flowing between the VMs and outside the host (e.g., when PCI-passthrough is used for NICs). In this case it would be a lot better for these VMs to use memory from different nodes and avoid wasting each other cache lines.

This project aims at making it possible for the Xen virtualization platform (intended as hypervisor + toolstack) to take advantage of this knowledge about the characteristics of the workload and use it to maximize performances. A first step would be to enhance the automatic NUMA placement algorithm to consider the cooperative-ness and/or the competitive-ness of a VM during placement itself, if provided with such information by the user. A much more complicated development could be to have this relationship between the various running VMs guessed automatically on-line (e.g., by watching the memory mappings and looking for specific patterns), and update the situation accordingly.

Trascendent memory (Tmem) can be seen as a mechanism for discriminating between frequently and infrequently used data, and thus helping allocating them properly. It would be interesting to investigate and implement all the necessary mechanisms to take advantage of this and improve performances of Tmem enabled guests running on NUMA machines.

Linux hotplug scripts should be analized, providing a good description of what each hotplug script is doing. After this, scripts should be cleaned, putting common pieces of code in shared files across all scripts. A Coding style should be applied to all of them when the refactoring is finished.

This project is to produce one or more command-line tools which support migrating VMs between these toolstacks.

One tool should be provided which takes an xl configuration file and details of an XCP pool. Using the XenAPI XML/RPC interface It should create a VM in the pool with a close approximation of the same configuration and stream the configured disk image into a selected Storage Repository.

A second tool should be provided which performs the opposite operation, i.e. give a reference to a VM residing in an XCP pool it should produce an XL compatible configuration file and stream the disk image(s) our of Xapi into a suitable format.

These tools could be reasonably bundled as part of either toolstack and by implication could be written in either C, Ocaml or some other suitable language.

The tool need not operate on a live VM but that could be considered a stretch goal.

An acceptable alternative to the proposed implementation would be to implement a tool which converts between a commonly used VM container format which is supported by XCP (perhaps OVF or similar) and the xl toolstack configuration file and disk image formats.

QEMU is capable of creating, listing and deleting disk snapshots on QCOW2 and QED files, so even today, issuing the right commands via the QEMU monitor, it is possible to create disk snapshots of a running Xen VM. xl and libxl don't have any knowledge of these snapshots, don't know how to create, list or delete them.

This project is about implementing disk snapshots support in libxl, using the QMP protocol to issue commands to QEMU. Users should be able to manage the entire life-cycle of their disk snapshots via xl. The candidate should also explore ways to integrate disk snapshots into the regular Xen save/restore mechanisms and provide a solid implementation for xl/libxl.

pass through GPUs to guests -- allowing people to run Linux as their main desktop but easily play games requiring proprietary operating systems without rebooting.

GPUs can be easily passed through to guests as secondary displays, but as of yet cannot be passed through as primary displays. The main reason is the lack of ability to load the VGA BIOS from the card into the guest.

The purpose of this project would be to allow HVM guests to load the physical card's VGA bios, so that the guest can boot with it as the primary display.

a number of users have requested the ability to encode more advanced scheduling logic. For instance, "Let this VM max out for 5 minutes out of any given hour; but after that, impose a cap of 20%, so that even if the system is idle he can't an unlimited amount of CPU power without paying for a higher level of service."

This is too coarse-grained to do inside the hypervisor; a user-space tool would be sufficient. The goal of this project would

kdd allows you to debug a running Windows virtual machine on Xen using standard Windows kernel debugging tools like WinDbg. kdd is an external debugger stub for the windows kernel. Windows can be debugged without enabling the debugger stub inside windows kernel by using kdd. This is important for debugging hard to reproduce problems on Windows virtual machines that may not have debugging enabled.

Expected Results:

1. Add support for Windows 8 (x86, x64) to kdd
2. Add support for Windows Server 2012 to kdd
3. Enhance kdd to allow WinDbg to write out usable Windows memory dumps (via .dump debugger extension) for all supported versions
4. Produce a user guide for kdd on Xen wiki page

To get this its as easy as having this in your guest config:

vcpus=2
maxvcpus=3

And when you launch the guest to play with 'xm vcpu-set 0 2', xm vcpu-set 0 3' and see the guest forget about one of the CPUs.

This is what you will see in the guest:

udevd-work[2421]: error opening ATTR{/sys/devices/system/cpu/cpu2/online} for writing: No such file or directory

If you instrument udev and look at the code you will find somebody came up with a "fix": http://serverfault.com/questions/329329/pv-ops-kernel-ignoring-cpu-hotplug-under-xen-4-domu

[    0.073006] WARNING: at /home/konrad/ssd/linux/kernel/rcutree.c:1547 __rcu_process_callbacks+0x42e/0x440()
[    0.073008] Modules linked in:
[    0.073010] Pid: 12, comm: migration/2 Not tainted 3.5.0-rc2 #2
[    0.073011] Call Trace:
[    0.073017]  <IRQ>  [<ffffffff810718ea>] warn_slowpath_common+0x7a/0xb0

which I get with this guest config:

vcpus=2
maxvcpus=3

I came up with a patch for the problem that William found: http://lists.xen.org/archives/html/xen-devel/2012-05/msg01963.html

and narrowed it down the Linux calling xen_set_pte with a PMD flag (so trying to setup a 2MB page). Currently the implemenation of xen_set_pte can't do 2MB but it will gladly accept the argument and the multicall will fail.

Peter did not like the x86 implemenation so I was thinking to implement some code in xen_set_pte that will detect that its a PMD flag and do "something". That something could be either probe the PTE's and see if there is enough space and if so just call the multicall 512 times, or perform a hypercall to setup a super-page. .. But then I wasn't sure how we would

and whether those patches are the right way or the bad way.

• OCaml: Mirage http://openmirage.org
• Haskell: HalVM https://github.com/GaloisInc/HaLVM#readme
• Erlang: ErlangOnXen http://erlangonxen.org
• Java: GuestVM http://labs.oracle.com/projects/guestvm/

Each of these is in a different state of reliability and usability. We would like to survey all of them, build some common representative benchmarks to evaluate them, and build a common toolchain based on XCP that will make it easier to share code across such efforts. This project will require a reasonable grasp of several programming languages and runtimes, and should be an excellent project to learn more about the innards of popular languages.

Existing attempts at integrating DRBD sit below the SM layer and thus do not enable one VBD per DRBD device. They also suffer from a split-brain situation that could be avoided by controlling active/standby status from XenAPI.

The XCP RRD daemon (xcp-rrdd) provides a plugin interface to allow other processes to provide data sources. In principle, these plugins can be written in any language by using the XML-RPC/JSON interface (although presently bindings only exist for OCaml).

The project: Create plugins that expose additional information, including things like:

• total amount of CPU cycles used by each VM
• VM- or VBD/VIF-level disk and network throughput
• number of event channels consumed per domain
• how much work is each VM demanding of qemu, netback, blkback, xenstored?
• perhaps other statistics only currently easily obtainable via xentrace

However, some data sources change very slowly (e.g. CPU temperature, available disk capacity). So it is overkill to sample them every five seconds. This becomes a problem when it is costly to sample them, perhaps because it involves a CPU-intensive computation or disk activity.

The RRD daemon provides a plugin interface to allow other processes to provide data sources.

• parse OpenFlow requests
• read OVS flow table(s)
• optionally maintain a whitelist of acceptable port/IP address tuples
• write OVS flow rules
• optionally act as a proxy between an OVS instance and an OpenFlow Controller, relaying requests which it has decided not to parse out to the Controller, and the results back to the OVS instance.

kernels as host and guest in the same installation.

• Task 1: Generalise the functions in osstest which generate debian-installer preseed files and manage the installation, to teach them how to set up PV and HVM guests, and provide an appropriate ts-* invocation script.
• Task 2: Extend the guest installer from task 1 to be able to install a kernel other than the one which comes from the Debian repository, so that it is possible to test one kernel as host with a different specified kernel as guest.
• Task 3: Determine which combinations of kernel branches should be added to the test schedules, push gates, etc. and write this up in a report for deployment by the infrastructure maintainers.
• More information: See xen-devel test reports. Code is at http://xenbits.xen.org/gitweb/?p=osstest.git;a=summary

• Task 1: Understand how best to automate installation of NetBSD. Write code in osstest which is able to automatically and noninteractively install NetBSD on a bare host.
• Task 2: Test and debug osstest's automatic building arrangements so that they can correctly build Xen on NetBSD.
• Task 3: Write code in osstest which can automatically install the Xen from task 2 on the system installed by task 1.
• Task 4: Debug at least one of the guest installation capabilities in osstest so that it works on the Xen system from task 3.
• Task 5: Rework the code from task 1 so that it can also install a NetBSD guest, ideally either as a guest of a Linux dom0 or of a NetBSD dom0.
• Task 6: Determine which versions of NetBSD and of Linux should be tested in which combinations and write this up in a report for deployment by the infrastructure maintainers.
• More information: See xen-devel test reports. Code is at http://xenbits.xen.org/gitweb/?p=osstest.git;a=summary