RHEL5 CentOS5 Xen Intel SR-IOV NIC Virtual Function VF PCI Passthru Tutorial

From Xen
Revision as of 17:35, 16 August 2012 by Pasik (talk | contribs)

Intel SR-IOV NIC Virtual Function (VF) PCI passthru with RHEL5/CentOS5 Xen


  • You need at least RHEL 5.8 / CentOS 5.8. Earlier EL5 versions have some bugs that prevent SR-IOV VF passthru from working properly. For example in EL 5.7 VF passthru works only once, and fails on the second time.
  • You need a system with hardware IOMMU for PCI passthru (Intel VT-d). IOMMU needs to be supported by the CPU, chipset, BIOS/firmware and Xen.

Components used in this tutorial:

  • Dell R510 server, BIOS version: 1.10.2 (04/27/2012).
  • Intel Xeon CPU L5640.
  • Virtualization technology (Intel VT-x/VMX) enabled in BIOS.
  • IOMMU/VT-d enabled in BIOS.
  • CentOS 5.8 x86_64 DVD1.
  • Stock Xen rpms from CentOS 5.8.
  • Stock kernel-xen from CentOS 5.8 as dom0 kernel.

EL5 Host installation and configuration

  • Install RHEL 5.8 or CentOS 5.8 x64 (64 bit) host using the "Server" profile from DVD1.
  • Use LVM disk configuration, and leave free space to the LVM volumegroup, so you can later create new LVM volumes and install VMs using LVM volumes as disks.
  • After installation disable SElinux in "/etc/selinux/config", change to "SELINUX=disabled".
  • Do all the usual network/IP, hostname, DNS etc configuration.
  • Make sure there is free unallocated space in the LVM volume group:
[root@dom0 ~]# vgdisplay
  --- Volume group ---
  VG Name               VolGroup00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  38
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                27
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               1.82 TB
  PE Size               32.00 MB
  Total PE              59551
  Alloc PE / Size       55670 / 1.70 TB
  Free  PE / Size       3881 / 121.28 GB
  VG UUID               QuouSc-MXj2-Hhdo-Ut9k-RLLO-jk9N-RjQD6E
  • Note the "Free PE / Size" line on the vgdisplay command output.
  • Update the system with "yum update".
  • Disable some of the extra services that are not needed in this tutorial:
chkconfig isdn off
chkconfig mcstrans off
chkconfig haldaemon off
chkconfig hidd off
chkconfig autofs off
chkconfig avahi-daemon off
chkconfig xfs off
chkconfig bluetooth off
chkconfig pcscd off
chkconfig iptables off
chkconfig ip6tables off
  • Install Xen and related packages:
yum install xen xen-libs kernel-xen libvirt virt-viewer python-virtinst xorg-x11-xauth
  • Edit "/etc/xen/xend-config.sxp" and disable (comment out) Xen network-script line:
#(network-script network-bridge)
  • We want to configure dom0 networking settings and bridges ourselves.
  • Modify "/boot/grub/grub.conf" and add all the usual options for Xen hypervisor:
title CentOS (2.6.18-308.11.1.el5xen)
        root (hd0,0)
        kernel /xen.gz-2.6.18-308.11.1.el5 dom0_mem=2048M loglvl=all
        module /vmlinuz-2.6.18-308.11.1.el5xen ro root=/dev/VolGroup00/LogVol00
        module /initrd-2.6.18-308.11.1.el5xen.img
  • Add "dom0_mem=2048M loglvl=all" options on the xen.gz line, adjust the dom0_mem as you wish/need.
  • Make sure the Xen entry is the default in Grub
  • Reboot the system to Xen.
  • When the system has booted up, verify Xen works OK:
[root@dom0 ~]# xm list
Name                                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                                   0     2048    12 r-----    788.0
  • Verify from the "xm list" output that dom0 is using the amount of memory you specified in Grub.
[root@dom0 ~]# xm info
host                   : dom0.localdomain
release                : 2.6.18-308.11.1.el5xen
version                : #1 SMP Tue Jul 10 09:29:47 EDT 2012
machine                : x86_64
nr_cpus                : 12
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 6
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps                : bfebfbff:2c100800:00000000:00000940:029ee3ff:00000000:00000001
total_memory           : 49139
free_memory            : 46145
node_to_cpu            : node0:0-11
xen_major              : 3
xen_minor              : 1
xen_extra              : .2-308.11.1.el5
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)
cc_compile_by          : mockbuild
cc_compile_domain      : centos.org
cc_compile_date        : Tue Jul 10 08:40:13 EDT 2012
xend_config_format     : 2


  • In this tutorial we're using the following Intel SR-IOV capable 10 Gbit/sec 82599 NIC: