Freenas with Xen

This howto is based on the excellent work made by My Wired House. It aims at building FreeNAS with PVHVM support and ultimately at putting it into production with PCI passthrough.

Build environment

• Install a Freebsd 9.3 (in a domU for example)
• Make a few adjustements
• Build Freenas

Freebsd 9.3 in a domU

Installation

• Option 1 : Xml configuration for Libvirt (Xen + Openvswitch)

<domain type='xen'>
  <name>freebsd</name>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='xenfv'>hvm</type>
    <loader type='rom'>/usr/lib/xen/boot/hvmloader</loader>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hap/>
  </features>
  <clock offset='variable' adjustment='0' basis='utc'>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/lib/xen/bin/qemu-system-i386</emulator>
    <disk type='block' device='disk'>
      <driver name='phy'/>
      <source dev='/dev/system/freebsd.reseau.ld-disk'/>
      <target dev='xvda' bus='xen'/>
    </disk>
    <disk type='file' device='cdrom'>
      <source file='/opt/ISO/FreeBSD-9.3-RELEASE-amd64-bootonly.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='ide' index='0'/>
    <interface type='bridge'>
      <mac address='00:16:3e:xx:xx:xx'/>
      <source bridge='xenbr0'/>
      <script path='/etc/xen/scripts/vif-openvswitch'/>
      <model type='e1000'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='fr'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1'/>
    </video>
  </devices>
</domain>

• Option 2 : cfg file for Xl

#
#  Kernel + memory size
#
builder      = 'hvm'
boot         = 'cd'
memory       = '2048'
vcpus        = '1'

#
#  Disk device(s).
#
disk        = [
                  'phy:/dev/system/freebsd.reseau.ld-disk,xvda,w', 'file:/opt/ISO/FreeBSD-9.3-RELEASE-amd64-bootonly.iso,hdc:cdrom,r'
              ]


#
#  Hostname
#
name        = 'freebsd'

#
#  Networking
#
vif         = [ 'mac=00:16:3e:xx:xx:xx, model=e1000' ]

#
#  Behaviour
#
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

sdl=0
vnc = 1
vncunused = 1
vnclisten = '0.0.0.0'
stdvga=0
serial='pty'
usb=1
usbdevice='tablet'
keymap='fr'
localtime=1

Update & PVHVM configuration (to boost performances)

# freebsd-update fetch
# freebsd-update install
# reboot
# cd /usr/src
# make buildkernel KERNCONF=XENHVM
# make installkernel KERNCONF=XENHVM
# sed -i".backup" 's/ada/xbd/g' /etc/fstab
# sed -i".backup" 's/ifconfig_em0/ifconfig_xn0/' /etc/rc.conf
# reboot

Install Freenas dependencies

Cf Readme on Github

# pkg install poudriere-devel git cdrtools pxz python grub2-pcbsd xorriso py27-sphinx py27-sphinxcontrib-httpdomain-1.2.1

Dowload Freenas 9.3 release

Dowload the last 9.3 Freenas release on Github (e.g. 9.3-STABLE-201506042008.tar.gz)

# mkdir -p /home && wget --no-check-certificate -O /home/9.3-STABLE-201506042008.tar.gz https://github.com/freenas/freenas/archive/9.3-STABLE-201506042008.tar.gz
# cd /home
# tar xf 9.3-STABLE-201506042008.tar.gz
# cd freenas-9.3-STABLE-201506042008
# make git-external

A few adjustements on Freenas sources

• Modify "VERSION" in Makefile to reflect the custom release :

# sed -i".backup" 's/VERSION\?=9.3-STABLE/VERSION\?=9.3-STABLE-xen/' /home/freenas-9.3-STABLE-201506042008/Makefile

• Edit /home/freenas-9.3-STABLE-201506042008/build/nanobsd-cfg/FREENAS.amd64 and at the end, add theses lines :

options         NO_ADAPTIVE_MUTEXES
options         NO_ADAPTIVE_RWLOCKS
options         NO_ADAPTIVE_SX

# Xen HVM support
options         XENHVM
device          xenpci

• Remove the "problematic" "set -e" in create_iso.sh (I know, this is bad but the compilation would stop at this stage if not ...) :

# sed -i".backup" 's/set -e/#set -e/' /home/freenas-9.3-STABLE-201506042008/build/create_iso.sh