RHEL5 CentOS5 Xen Intel SR-IOV NIC Virtual Function VF PCI Passthru Tutorial

From Xen
Revision as of 17:47, 16 August 2012 by Pasik (talk | contribs) (Added SR-IOV configuration stuff)

Intel SR-IOV NIC Virtual Function (VF) PCI passthru with RHEL5/CentOS5 Xen


  • You need at least RHEL 5.8 / CentOS 5.8. Earlier EL5 versions have some bugs that prevent SR-IOV VF passthru from working properly. For example in EL 5.7 VF passthru works only once, and fails on the second time.
  • You need a system with hardware IOMMU for PCI passthru (Intel VT-d). IOMMU needs to be supported by the CPU, chipset, BIOS/firmware and Xen.

Components used in this tutorial:

  • Dell R510 server, BIOS version: 1.10.2 (04/27/2012).
  • Intel Xeon CPU L5640.
  • Virtualization technology (Intel VT-x/VMX) enabled in BIOS.
  • IOMMU/VT-d enabled in BIOS.
  • CentOS 5.8 x86_64 DVD1.
  • Stock Xen rpms from CentOS 5.8.
  • Stock kernel-xen from CentOS 5.8 as dom0 kernel.

EL5 Host installation and configuration

  • Install RHEL 5.8 or CentOS 5.8 x64 (64 bit) host using the "Server" profile from DVD1.
  • Use LVM disk configuration, and leave free space to the LVM volumegroup, so you can later create new LVM volumes and install VMs using LVM volumes as disks.
  • After installation disable SElinux in "/etc/selinux/config", change to "SELINUX=disabled".
  • Do all the usual network/IP, hostname, DNS etc configuration.
  • Make sure there is free unallocated space in the LVM volume group:
[root@dom0 ~]# vgdisplay
  --- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  38
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                27
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.82 TB
  PE Size               32.00 MB
  Total PE              59551
  Alloc PE / Size       55670 / 1.70 TB
  Free  PE / Size       3881 / 121.28 GB
  VG UUID               QuouSc-MXj2-Hhdo-Ut9k-RLLO-jk9N-RjQD6E
  • Note the "Free PE / Size" line on the vgdisplay command output above.
  • Update the system with "yum update".
  • Disable some of the extra services that are not needed in this tutorial:
chkconfig isdn off
chkconfig mcstrans off
chkconfig haldaemon off
chkconfig hidd off
chkconfig autofs off
chkconfig avahi-daemon off
chkconfig xfs off
chkconfig bluetooth off
chkconfig pcscd off
chkconfig iptables off
chkconfig ip6tables off
  • Install Xen and related packages:
yum install xen xen-libs kernel-xen libvirt virt-viewer python-virtinst xorg-x11-xauth
  • Edit "/etc/xen/xend-config.sxp" and disable (comment out) Xen network-script line:
#(network-script network-bridge)
  • We want to configure dom0 networking settings and bridges ourselves.
  • Modify "/boot/grub/grub.conf" and add all the usual options for Xen hypervisor:
title CentOS (2.6.18-308.11.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-308.11.1.el5 dom0_mem=2048M loglvl=all
        module /vmlinuz-2.6.18-308.11.1.el5xen ro root=/dev/VolGroup00/LogVol00
        module /initrd-2.6.18-308.11.1.el5xen.img
  • Add "dom0_mem=2048M loglvl=all" options on the xen.gz line, adjust the dom0_mem as you wish/need.
  • Make sure the Xen entry is the default in Grub
  • Reboot the system to Xen.
  • When the system has booted up, verify Xen works OK:
[root@dom0 ~]# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     2048    12 r-----    788.0
  • Verify from the "xm list" output that dom0 is using the amount of memory you specified in Grub.
[root@dom0 ~]# xm info
host                   : dom0.localdomain
release                : 2.6.18-308.11.1.el5xen
version                : #1 SMP Tue Jul 10 09:29:47 EDT 2012
machine                : x86_64
nr_cpus                : 12
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps                : bfebfbff:2c100800:00000000:00000940:029ee3ff:00000000:00000001
total_memory           : 49139
free_memory            : 46145
node_to_cpu            : node0:0-11
xen_major              : 3
xen_minor              : 1
xen_extra              : .2-308.11.1.el5
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)
cc_compile_by          : mockbuild
cc_compile_domain      : centos.org
cc_compile_date        : Tue Jul 10 08:40:13 EDT 2012
xend_config_format     : 2


  • In this tutorial we're using the following Intel SR-IOV capable 10 Gbit/sec 82599 NIC:
[root@dom0 ~]# lspci | grep 10-Gig
03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 01)
  • which uses the Intel ixgbe driver, as can be seen from the "ethtool -i" output:
[root@dom0 ~]# ethtool -i eth0
driver: ixgbe
version: 3.4.8-k
firmware-version: 0.9-3
bus-info: 0000:03:00.0
[root@dom0 ~]# ethtool -i eth1
driver: ixgbe
version: 3.4.8-k
firmware-version: 0.9-3
bus-info: 0000:03:00.1

Enabling IOMMU support in Xen command line options

  • Edit "/boot/grub/grub.conf" and add "iommu=1" option for Xen, and "pci_pt_e820_access=on" option for dom0 Linux kernel (vmlinuz). Both options are required:
title CentOS (2.6.18-308.11.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-308.11.1.el5 dom0_mem=2048M loglvl=all iommu=1
        module /vmlinuz-2.6.18-308.11.1.el5xen ro root=/dev/VolGroup00/LogVol00 pci_pt_e820_access=on
        module /initrd-2.6.18-308.11.1.el5xen.img

Enable Intel SR-IOV NIC Virtual Functions (VFs) in the ixgbe driver options

  • Add the following line to end of "/etc/modprobe.conf":
options ixgbe max_vfs=8
  • 8 is just an example here, 82599 NIC supports up to 64 VFs.
  • Blacklist the Intel VF driver (ixgbevf) in dom0 so that dom0 kernel doesn't try to use the Virtual Functions (we want to PCI passthru them to Xen VMs!). Create "/etc/modprobe.d/blacklist-ixgbevf.conf" file with the following contents:
# intel ixgbe sr-iov vf (virtual function) driver
blacklist ixgbevf
  • Now reboot the system to activate the changes.
  • After reboot use "lspci" to verify you can see VFs: