RHEL5 CentOS5 Xen Intel SR-IOV NIC Virtual Function VF PCI Passthru Tutorial

From Xen
Revision as of 17:28, 16 August 2012 by Pasik (talk | contribs) (Initial version, contains only basic system configuration, no SR-IOV stuff yet.)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Intel SR-IOV NIC Virtual Function (VF) PCI passthru with RHEL5/CentOS5 Xen


  • You need at least RHEL 5.8 / CentOS 5.8. Earlier EL5 versions have some bugs that prevent SR-IOV VF passthru from working properly. For example in EL 5.7 VF passthru works only once, and fails on the second time.
  • You need a system with hardware IOMMU for PCI passthru (Intel VT-d). IOMMU needs to be supported by the CPU, chipset, BIOS/firmware and Xen.

Components used in this tutorial:

  • Dell R510 server, BIOS version: 1.10.2 (04/27/2012).
  • Intel Xeon CPU L5640.
  • Virtualization technology (Intel VT-x/VMX) enabled in BIOS.
  • IOMMU/VT-d enabled in BIOS.
  • CentOS 5.8 x86_64 DVD1.
  • Stock Xen rpms from CentOS 5.8.
  • Stock kernel-xen from CentOS 5.8 as dom0 kernel.

EL5 Host installation and configuration

  • Install RHEL 5.8 or CentOS 5.8 x64 (64 bit) host using the "Server" profile from DVD1.
  • Use LVM disk configuration, and leave free space to the LVM volumegroup, so you can later create new LVM volumes and install VMs using LVM volumes as disks.
  • After installation disable SElinux in "/etc/selinux/config", change to "SELINUX=disabled".
  • Do all the usual network/IP, hostname, DNS etc configuration.
  • Update the system with "yum update".
  • Disable some of the extra services that are not needed in this tutorial:
chkconfig isdn off
chkconfig mcstrans off
chkconfig haldaemon off
chkconfig hidd off
chkconfig autofs off
chkconfig avahi-daemon off
chkconfig xfs off
chkconfig bluetooth off
chkconfig pcscd off
chkconfig iptables off
chkconfig ip6tables off
  • Install Xen and related packages:
yum install xen xen-libs kernel-xen libvirt virt-viewer python-virtinst xorg-x11-xauth
  • Edit "/etc/xen/xend-config.sxp" and disable Xen network-script line:
#(network-script network-bridge)
  • We want to configure dom0 networking settings and bridges ourselves.
  • Modify "/boot/grub/grub.conf" and add all the usual options for Xen hypervisor:
title CentOS (2.6.18-308.11.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-308.11.1.el5 dom0_mem=2048M loglvl=all
        module /vmlinuz-2.6.18-308.11.1.el5xen ro root=/dev/VolGroup00/LogVol00
        module /initrd-2.6.18-308.11.1.el5xen.img
  • Add "dom0_mem=2048M loglvl=all" options on the xen.gz line, adjust the dom0_mem as you wish/need.
  • Make sure the Xen entry is the default in Grub
  • Reboot the system to Xen.
  • When the system has booted up, verify Xen works OK:
[root@dom0 ~]# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     2048    12 r-----    788.0
  • Verify from the "xm list" output that dom0 is using the amount of memory you specified in Grub.
[root@dom0 ~]# xm info
host                   : dom0.localdomain
release                : 2.6.18-308.11.1.el5xen
version                : #1 SMP Tue Jul 10 09:29:47 EDT 2012
machine                : x86_64
nr_cpus                : 12
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps                : bfebfbff:2c100800:00000000:00000940:029ee3ff:00000000:00000001
total_memory           : 49139
free_memory            : 46145
node_to_cpu            : node0:0-11
xen_major              : 3
xen_minor              : 1
xen_extra              : .2-308.11.1.el5
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)
cc_compile_by          : mockbuild
cc_compile_domain      : centos.org
cc_compile_date        : Tue Jul 10 08:40:13 EDT 2012
xend_config_format     : 2