In my last post I wrote about to getting my Pi2 to boot with HYP enabled on all 4 CPUs. The next stage is to get a kernel with KVM enabled and get a VM up and running. Once again most of this method is taken from a blog post by @sergiolpascual{.twitter-atreply.pretty-link} I have tidied it up and refined the method of using a single CPU core without patching QEMU.

Building a KVM enabled kernel for the Pi

First of all you need to get a KVM enabled kernel for the Pi2 host. These commands checkout the current 3.18 version of the Raspberry Pi foundation’s kernel tree and apply a pull request on top of it to enable GIC emulation.

git clone https://github.com/raspberrypi/linux
cd linux
git checkout rpi-3.18.y
git fetch origin pull/902/head:VGIC-emu
git checkout VGIC-emu

I started off with the same base config as the kernel on my Pi 2. There is a file /proc/config.gz that stores the config the running kernel was built with. Copy this file to your build host, uncompress it and rename it to /working/rpi2/.config that way you will only need to make a few minor changes to the config.

scp user@pi:/proc/config.gz /tmp/config.gz
mkdir -p /working/rpi2
zcat /tmp/config.gz > /working/rpi2/.config

Now we need to setup a cross compile environment again and start the kernel config system.

export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-
make O=/working/rpi2 menuconfig

I’ve listed the options that need to be changed to enable KVM support in the kernel, these notes are copied from @sergiolpascual{.twitter-atreply.pretty-link}‘s post. I have uploaded my KVM enabled config with these options set already.

Note the O= option this sets the directory where compiled parts of the kernel are put, it means that your source tree stay clean.

  • Patch physical to virtual translations at runtime
  • General setup -> Control Group support
  • System Type -> Support for Large Physical Address Extension
  • Boot options -> Use appended device tree blob to zImage (EXPERIMENTAL)
  • Boot options -> Supplement the appended DTB with traditional ATAG information
  • Virtualization -> Kernel-based Virtual Machine (KVM) support (NEW)
    • DISABLE Virtualization -> KVM support for Virtual GIC
    • ENABLE Virtualization -> KVM support for Emulated GIC

Now you just need to build the kernel and install the modules

make O=/working/rpi2 all -j 4
export INSTALL_MOD_PATH=/tmp/rpi-kernel
make O=/working/rpi2 modules_install

To build the Pi2 boot image, glue the kernel and DTB file together thus

cat /working/rpi2/arch/arm/boot/zImage /working/rpi2/arch/arm/boot/dts/bcm2709-rpi-2-b.dtb > /tmp/rpi-kernel/kernel7.img

Copy the directory /tmp/rpi-kernel to the Pi2, replace /boot/firmware/kernel7.img (take a backup first) with our new kernel and move the new 3.18.x modules directory to /lib/modules/.

You will need to adjust the Pi’s kernel command line to add “isolcpus=3” to work round a bug. To do this via u-boot run these commands from the u-boot command line:

setenv "${bootargs} isolcpus=3"
saveenv

A final reboot and you should get this in dmesg

dmesg | grep kvm
kvm [1]: timer IRQ99
kvm [1]: Hyp mode initialized successfully

Booting your first Virtual Machine

So long as you are running Debian / Raspbian Jessie then you can just run

apt-get install qemu-system

Adding the boot option isolcpus=3 works round an oddity of the Raspberry Pi’s CPU, discussed in more detail in the original blog post. We need to ensure that QEMU only runs on this isolated CPU. In the original post this was done by patching QEMU but there is a much easier way, taskset. Taskset allows us to restrict QEMU to only CPU 3 with the “-c 3-3” option.

This is my basic run script to boot an ARM VM on the Raspberry Pi2. I used the Linaro prebuilt ARM kernels and root images to test with and have included the URLs in the script.

#!/bin/sh

# Disable the QEMU sound driver
export QEMU_AUDIO_DRV=none

# Basic system setup an ARM vexpress with 1 CPU, 256M of RAM
smp=1
cpu=host
ram=256
machine=vexpress-a15
# Where are the kernel and root images stored
dir=/root/linaro

# Source: https://snapshots.linaro.org/ubuntu/images/kvm/latest/zImage-armv7 
kernel=$dir/zImage-armv7
# Source: https://snapshots.linaro.org/ubuntu/images/kvm/latest/vexpress-v2p-ca15-tc1.dtb 
dtb=$dir/vexpress-v2p-ca15-tc1.dtb
# Source: http://snapshots.linaro.org/ubuntu/images/kvm-guest/36/armhf/kvm-armhf.qcow2.xz
rootfs=$dir/kvm-armhf.qcow2
# Virtual machine Linux command line
cmdline="root=/dev/vda2 rw mem=${ram}M console=ttyAMA0 rootwait rootfstype=ext4"

# Use taskset to ensure that QEMU only runs on cpu 3
taskset -c 3-3 qemu-system-arm -enable-kvm -smp $smp -m $ram -M $machine -cpu host -kernel $kernel -dtb $dtb -append "$cmdline" -drive if=none,id=rootfs,file=$rootfs -device virtio-blk-device,drive=rootfs -netdev user,id=user0 -device virtio-net-device,netdev=user0 -nographic