Freenas with Xen
Sauter à la navigation
Sauter à la recherche
This howto is based on the excellent work made by My Wired House. It aims at building FreeNAS with Xen PVHVM support and ultimately at putting it into production with PCI passthrough.
Build environment
- Install Freebsd 9.3 (in a domU for example)
- Make a few adjustements
- Build Freenas
Installation of Freebsd 9.3 in a domU
- Install Freebsd 9.3 on a disk / volume of 100 Go (building Frenas requires space !)
- Configuration file for Xl
#
# Kernel + memory size
#
builder = 'hvm'
boot = 'cd'
memory = '2048'
vcpus = '1'
#
# Disk device(s).
#
disk = [
'phy:/dev/system/freebsd-disk,xvda,w',
'file:/opt/ISO/FreeBSD-9.3-RELEASE-amd64-bootonly.iso,hdc:cdrom,r'
]
#
# Hostname
#
name = 'freebsd'
#
# Networking
#
vif = [ 'mac=00:16:3e:xx:xx:xx, model=e1000' ]
#
# Behaviour
#
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
sdl=0
vnc = 1
vncunused = 1
vnclisten = '0.0.0.0'
stdvga=0
serial='pty'
usb=0
keymap='fr'
localtime=1
- Note: To speed up the process, add RAM and vCPUs if you can !
Update & PVHVM configuration (to boost performances)
# freebsd-update fetch
# freebsd-update install
# reboot
# cd /usr/src
# make buildkernel KERNCONF=XENHVM
# make installkernel KERNCONF=XENHVM
# sed -i".backup" 's/ada/xbd/g' /etc/fstab
# sed -i".backup" 's/ifconfig_em0/ifconfig_xn0/' /etc/rc.conf
# reboot
Building Freenas
Install Freenas dependencies
# pkg install poudriere-devel git cdrtools pxz python grub2-pcbsd xorriso py27-sphinx py27-sphinxcontrib-httpdomain-1.2.1
Dowload Freenas 9.3.1 release
Dowload the latest Freenas 9.3.1 release on Github (e.g. 9.3.1-STABLE-201508250051.tar.gz)
# mkdir -p /home && wget --no-check-certificate -O /home/9.3.1-STABLE-201508250051.tar.gz https://github.com/freenas/freenas/archive/9.3.1-STABLE-201508250051.tar.gz
# cd /home
# tar xf 9.3.1-STABLE-201508250051.tar.gz
# mv 9.3.1-STABLE-201508250051 freenas
# cd freenas
A few adjustements on Freenas sources
- Modify "VERSION" in Makefile to reflect the custom release :
# sed -i".backup" 's/VERSION\?=9.3.1-STABLE/VERSION\?=9.3.1-STABLE-xen/' /home/freenas/Makefile
# sed -i".backup" 's/TRAIN\?=\${NANO_LABEL}-9.3-STABLE/TRAIN\?=${NANO_LABEL}-9.3-STABLE-xen/' /home/freenas/Makefile
- Edit /home/freenas/build/nanobsd-cfg/FREENAS.amd64 and at the end, add theses lines :
options NO_ADAPTIVE_MUTEXES
options NO_ADAPTIVE_RWLOCKS
options NO_ADAPTIVE_SX
# Xen HVM support
options XENHVM
device xenpci
- Remove the "problematic" "set -e" in create_iso.sh (I know, this is bad but the process would stop at this stage if not ...) :
# sed -i".backup" 's/set -e/#set -e/' /home/freenas/build/create_iso.sh
Build !
Depending on the ressources allocated to your Freebsd build machine, it will take a while !
# make checkout
# env _KEY=set NANO_LABEL=FreeNAS VERSION=9.3-STABLE-xen \
GIT_LOCATION=EXTERNAL \
BUILD_TIMESTAMP=`date '+%Y%m%d%H%M'` \
SEQUENCE=FreeNAS-9.3-STABLE-xen-201506042008 \
TRAIN=FreeNAS-9.3-STABLE-xen \
UPDATE_USER=sef FREENAS_KEYFILE=Keys/ix-nightly-key.key \
CHANGELOG=ChangeLog \
make release
If all went well, you will find the result in /home/freenas/objs/FreeNAS-9.3-STABLE-xen-201506052152/x64 :
# ls /home/freenas/objs/FreeNAS-9.3-STABLE-xen-201506052152/x64
FreeNAS-9.3-STABLE-xen-201506052152.GUI_Upgrade.txz FreeNAS-9.3-STABLE-xen-201506052152.iso
FreeNAS-9.3-STABLE-xen-201506052152.GUI_Upgrade.txz.sha256.txt FreeNAS-9.3-STABLE-xen-201506052152.iso.sha256.txt
Installation of Freenas 9.3 in a domU
Installation is done through the following steps :
- boot of the installer with special option hint.hpet.0.clock=0 and without PCI passthrough
- first boot of the fresh Freenas installation with special option hint.hpet.0.clock=0 (yes again ...) and without PCI passthrough
- in Freenas, adding the necessary options for the bootloader and PCI passthrgouh ... And the last reboot with PCI passthrough activated
Xl config file
#
# Kernel + memory size
#
builder = 'hvm'
boot = 'cd'
memory = '8192'
vcpus = '2'
#
# Disk device(s).
#
disk = [
'phy:/dev/system/freenas-disk,xvda,w',
'file:/opt/ISO/FreeNAS-9.3-STABLE-xen-201506052152.iso,hdb:cdrom,r'
]
#
# Hostname
#
name = 'freenas'
#
# Networking
#
vif = [ 'mac=00:16:3e:xx:xx:xx, model=e1000' ]
#
# Behaviour
#
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
sdl=0
vnc = 1
vncunused = 1
vnclisten = '0.0.0.0'
stdvga=0
serial='pty'
usb=0
keymap='fr'
localtime=1
Booting the installer
Because of this little issue, booting the installer requires special option hint.hpet.0.clock=0
- Start the vm
# xl create /etc/xen/freenas.ld
- As soon as the VM boots, open the console (e.g. via VNC), go to the Grub command prompt by pressing "c" and enter the following lines :
Grub options to start the installer
- Follow the steps to install Freenas ; note the name of the volume (xbd0) which reflects the use of PV drivers
Booting your fresh Freenas installation
Once again, because of this little issue, booting the fresh Freenas installation for the first time requires special option hint.hpet.0.clock=0
- As soon as the VM reboots after Freenas installation, reopen the console (e.g. via VNC), go to the Grub command prompt by pressing "c" and enter the following lines :
grub> insmod zfs
grub> kfreebsd /ROOT/default/@/boot/kernel/kernel
grub> kfreebsd_loadenv /ROOT/default@/boot/device.hints
grub> set kFreeBSD.kernelname="/boot/kernel/kernel"
grub> kfreebsd_module /ROOT/default/@/boot/zfs/rpool.cache type=/boot/zfs/zpool.cache
grub> set kFreeBSD.vfs.root.mountfrom=zfs:freenas-boot/ROOT/default
grub> kfreebsd_module_elf /ROOT/default/@/boot/kernel/ispfw.ko
grub> set kFreeBSD.hint.hpet.0.clock=0
grub> boot
- In Freenas GUI, go to "System" -> "Tunales" -> "Add Tunable" :
- Variable : hint.hpet.0.clock
- Value : 0
- Type : loader
- For PCI passthrough with LSI devices such as LSI 9211-8i (LSI 9211 cards in IT mode are recommanded with Freenas), two more tunables are required ; in Freenas GUI, go to "System" -> "Tunables" -> "Add Tunable" :
- Variable : hw.pci.enable_msi
- Value : 1
- Type : sysctl
- Variable : hw.pci.enable_msix
- Value : 0
- Type : sysctl
