Freenas with Xen

note : this tutorial applies to Freenas 9.3. Feenas 9.10 (based on FreeBSD 10) and upward natively includes the optimizations for Xen and LSI cards described here.

This howto is based on the excellent work made by My Wired House. It aims at building FreeNAS with Xen PVHVM support and ultimately at putting it into production with PCI passthrough.

Build environment

• Install Freebsd 9.3 (in a domU for example)
• Make a few adjustements
• Build Freenas

Installation of Freebsd 9.3 in a domU

• Install Freebsd 9.3 on a disk / volume of 100 Go (building Frenas requires space !)

• Configuration file for Xl

#
#  Kernel + memory size
#
builder      = 'hvm'
boot         = 'cd'
memory       = '2048'
vcpus        = '1'

#
#  Disk device(s).
#
disk        = [
                  'phy:/dev/system/freebsd-disk,xvda,w', 
                  'file:/opt/ISO/FreeBSD-9.3-RELEASE-amd64-bootonly.iso,hdc:cdrom,r'
              ]


#
#  Hostname
#
name        = 'freebsd'

#
#  Networking
#
vif         = [ 'mac=00:16:3e:xx:xx:xx, model=e1000' ]

#
#  Behaviour
#
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

sdl=0
vnc = 1
vncunused = 1
vnclisten = '0.0.0.0'
stdvga=0
serial='pty'
usb=0
keymap='fr'
localtime=1

• Note: To speed up the process, add RAM and vCPUs if you can !

Update & PVHVM configuration (to boost performances)

freebsd-update fetch
freebsd-update install
reboot
cd /usr/src
make buildkernel KERNCONF=XENHVM
make installkernel KERNCONF=XENHVM
sed -i".backup" 's/ada/xbd/g' /etc/fstab
sed -i".backup" 's/ifconfig_em0/ifconfig_xn0/' /etc/rc.conf
reboot

Building Freenas

Install Freenas dependencies

Cf Readme on Github

pkg install poudriere-devel git cdrtools pxz python grub2-pcbsd xorriso py27-sphinx py27-sphinxcontrib-httpdomain-1.2.1

Clone Freenas 9.3-STABLE branch

Clone the Freenas 9.3-STABLE branch from Github

mkdir /home
cd /home && git clone https://github.com/freenas/freenas.git
cd freenas && git checkout 9.3-STABLE
make checkout

A few adjustements on Freenas sources

• Modify "VERSION" in Makefile to reflect the custom release :

sed -i".backup" 's/VERSION\?=9.3.1-STABLE/VERSION\?=9.3.1-STABLE-xen/ ; s/TRAIN\?=\${NANO_LABEL}-9.3-STABLE/TRAIN\?=${NANO_LABEL}-9.3-STABLE-xen/' /home/freenas/Makefile

• Edit /home/freenas/build/nanobsd-cfg/FREENAS.amd64 and at the end, add theses lines :

options         NO_ADAPTIVE_MUTEXES
options         NO_ADAPTIVE_RWLOCKS
options         NO_ADAPTIVE_SX

# Xen HVM support
options         XENHVM
device          xenpci

• Remove the "problematic" "set -e" in create_iso.sh (I know, this is bad but the process would stop at this stage if not ...) :

sed -i".backup" 's/set -e/#set -e/' /home/freenas/build/create_iso.sh

Build !

Depending on the ressources allocated to your Freebsd build machine, it will take a while !

make release

If all went well, you will find the result in /home/freenas/objs/FreeNAS-9.3-STABLE-xen-201506052152/x64 :

ls /home/freenas/objs/FreeNAS-9.3.1-STABLE-xen-201509111019/x64
FreeNAS-9.3-STABLE-xen-201509111019.GUI_Upgrade.txz             FreeNAS-9.3-STABLE-xen-201509111019.iso
FreeNAS-9.3-STABLE-xen-201509111019.GUI_Upgrade.txz.sha256.txt  FreeNAS-9.3-STABLE-xen-201509111019.iso.sha256.txt

Installation of Freenas 9.3 in a domU

Installation is done through the following steps :

• boot of the installer without PCI passthrough
• first boot of the fresh Freenas installation without PCI passthrough
• in Freenas, adding the necessary options for PCI passthrgouh ... And the last reboot with PCI passthrough activated

Xl config file

#
#  Kernel + memory size
#
builder      = 'hvm'
boot         = 'cd'
memory       = '8192'
vcpus        = '2'

#
#  Disk device(s).
#
disk        = [
                  'phy:/dev/system/freenas-disk,xvda,w',
                  'file:/opt/ISO/FreeNAS-9.3-STABLE-xen-201506052152.iso,hdb:cdrom,r'
              ]


#
#  Hostname
#
name        = 'freenas'

#
#  Networking
#
vif         = [ 'mac=00:16:3e:xx:xx:xx, model=e1000' ]

#
#  Behaviour
#
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

sdl=0
vnc = 1
vncunused = 1
vnclisten = '0.0.0.0'
stdvga=0
serial='pty'
usb=0
keymap='fr'
localtime=1

Booting the installer

• Start the vm

xl create /etc/xen/freenas.ld

• Follow the steps to install Freenas ; choose the volume named xbd0 which reflects the use of PV drivers

Booting your fresh Freenas installation

For PCI passthrough with LSI devices such as LSI 9211-8i (LSI 9211 cards in IT mode are recommanded with Freenas), two more tunables are required ; in Freenas GUI, go to "System" -> "Tunables" -> "Add Tunable" :

• Variable : hw.pci.enable_msi
  • Value : 1
  • Type : sysctl
• Variable : hw.pci.enable_msix
  • Value : 0
  • Type : sysctl