Set up xhyve with Ubuntu 16.04

README.md

Introduction

Setting up an OS to work with xhyve can be a bit tricky, but it's not imppossible. These intructions should be generally applicable to most versions of Linux.

Create a disk image file

Xhyve will use a file as a logical disk. Be sure you have the filesize you need, because growing the file later is tricky to impossible (you could try to use qcow or similar to get around this, but qcow volumes can be tricky to mount).

dd if=/dev/zero of=hdd.img bs=1g count=32 # Create a 32 gig disk. Raise 'count' as desired

Grab required kernel files

Xhyve loads the kernel and initrd images from the local filesystem -- you'll need to get these from a mounted iso in order to boot an install cd. First download an iso. You probably want to use the mini.iso.

Once downloaded, you'll need mount it and copy off the kernel files. But these isos are not directly mountable in osx. To make them mountable, you'll need to make a compatible copy and then mount the copy:

dd if=/dev/zero bs=2k count=1 of=tmp.iso
dd if=mini.iso bs=2k skip=1 >> tmp.iso
hdiutil attach tmp.iso

Next copy the kernel files from the mounted volume

cp /Volumes/CDROM/linux .
cp /Volumes/CDROM/initrd.gz .

You can now boot into an installer. Run the attached install.sh script. Follow the prompts. However, once tghe installation is complete -- do not reboot. Select Go Back and then Execute a shell. We are going to use nc to copy off the system inird and linux kernel. At the shell prompt:

cd /target
sbin/ifconfig
tar c boot | nc -l -p 1234

Make note of the ip returned from ifconfig. On your osx host:

 nc <IP from above> 1234 | tar x

This should drop the latest boot images (kernel, initrd) into the boot/ subdirectory.

Exit the shell, and let the installation complete.

You should be able to boot up the new machine using the run.sh script below. Note that you will probably need to replace the exact filenames used for LINUX and INITRD with whatever you downloaded in boot/ in the previous step. The console is a little wonky -- in general, I install openssh when I am installing ubuntu, and use that to finish setting up the new vm after booting it up.

VPNs

The VM can lose connectivity to any VPNs running on the host after those vpns have been restarted, or after sleeping. Use the attached masq.sh to restore connectivity when this happens.

install.sh

#!/bin/bash
KERNEL="linux"
INITRD="initrd.gz"
CMDLINE="earlyprintk=serial console=ttyS0"

# Guest Config
MEM="-m 1G"
IMG_CD="-s 1,ahci-cd,mini.iso"
IMG_HDD="-s 2,virtio-blk,hdd.img"
NET="-s 3:0,virtio-net,en0"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
LPC_DEV="-l com1,stdio"
ACPI="-A"

# and now run
sudo xhyve $ACPI $MEM $PCI_DEV $LPC_DEV $NET $IMG_CD $IMG_HDD -f kexec,$KERNEL,$INITRD,"$CMDLINE"

masq.sh

#!/bin/bash
interfaces=( $(netstat -in | egrep 'utun\d .*\d+\.\d+\.\d+\.\d+' | cut -d ' ' -f 1) )
rulefile="rules.tmp"
echo "" > $rulefile
sudo pfctl -a com.apple/tun -F nat
for i in "${interfaces[@]}"
do
  RULE="nat on ${i} proto {tcp, udp, icmp} from 192.168.64.0/24 to any -> ${i}"
  echo $RULE >> $rulefile
done
sudo pfctl -a com.apple/tun -f $rulefile

run.sh

KERNEL="boot/vmlinuz-4.4.0-21-generic"
INITRD="boot/initrd.img-4.4.0-21-generic"
CMDLINE="earlyprintk=serial console=ttyS0 acpi=off root=/dev/vda1 ro"
UUID="-U 8e7af180-c54d-4aa2-9bef-59d94a1ac572" # A UUID will ensure we get a consistent ip address assigned
# Guest Config
MEM="-m 3G"
IMG_HDD="-s 2,virtio-blk,hdd.img"
NET="-s 3:0,virtio-net,en0"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
LPC_DEV="-l com1,stdio"
ACPI="-A"

# and now run
sudo xhyve $UUID $ACPI $MEM $PCI_DEV $LPC_DEV $NET $IMG_CD $IMG_HDD -f kexec,$KERNEL,$INITRD,"$CMDLINE"