In my last post I wrote about to getting my Pi2 to boot with HYP enabled on all 4 CPUs. The next stage is to get a kernel with KVM enabled and get a VM up and running. Once again most of this method is taken from a blog post by @sergiolpascual{.twitter-atreply.pretty-link} I have tidied it up and refined the method of using a single CPU core without patching QEMU.
Building a KVM enabled kernel for the Pi
First of all you need to get a KVM enabled kernel for the Pi2 host. These commands checkout the current 3.18 version of the Raspberry Pi foundation’s kernel tree and apply a pull request on top of it to enable GIC emulation.
git clone https://github.com/raspberrypi/linux cd linux git checkout rpi-3.18.y git fetch origin pull/902/head:VGIC-emu git checkout VGIC-emu
I started off with the same base config as the kernel on my Pi 2. There is a file /proc/config.gz that stores the config the running kernel was built with. Copy this file to your build host, uncompress it and rename it to /working/rpi2/.config that way you will only need to make a few minor changes to the config.
scp user@pi:/proc/config.gz /tmp/config.gz mkdir -p /working/rpi2 zcat /tmp/config.gz > /working/rpi2/.config
Now we need to setup a cross compile environment again and start the kernel config system.
export ARCH=arm export CROSS_COMPILE=arm-linux-gnueabihf- make O=/working/rpi2 menuconfig
I’ve listed the options that need to be changed to enable KVM support in the kernel, these notes are copied from @sergiolpascual{.twitter-atreply.pretty-link}‘s post. I have uploaded my KVM enabled config with these options set already.
Note the O= option this sets the directory where compiled parts of the kernel are put, it means that your source tree stay clean.
Patch physical to virtual translations at runtime
General setup -> Control Group support
System Type -> Support for Large Physical Address Extension
Boot options -> Use appended device tree blob to zImage (EXPERIMENTAL)
Boot options -> Supplement the appended DTB with traditional ATAG information
Virtualization -> Kernel-based Virtual Machine (KVM) support (NEW)
- DISABLE
Virtualization -> KVM support for Virtual GIC
- ENABLE
Virtualization -> KVM support for Emulated GIC
- DISABLE
Now you just need to build the kernel and install the modules
make O=/working/rpi2 all -j 4 export INSTALL_MOD_PATH=/tmp/rpi-kernel make O=/working/rpi2 modules_install
To build the Pi2 boot image, glue the kernel and DTB file together thus
cat /working/rpi2/arch/arm/boot/zImage /working/rpi2/arch/arm/boot/dts/bcm2709-rpi-2-b.dtb > /tmp/rpi-kernel/kernel7.img
Copy the directory /tmp/rpi-kernel to the Pi2, replace /boot/firmware/kernel7.img (take a backup first) with our new kernel and move the new 3.18.x modules directory to /lib/modules/.
You will need to adjust the Pi’s kernel command line to add “isolcpus=3” to work round a bug. To do this via u-boot run these commands from the u-boot command line:
setenv "${bootargs} isolcpus=3" saveenv
A final reboot and you should get this in dmesg
dmesg | grep kvm kvm [1]: timer IRQ99 kvm [1]: Hyp mode initialized successfully
Booting your first Virtual Machine
So long as you are running Debian / Raspbian Jessie then you can just run
apt-get install qemu-system
Adding the boot option isolcpus=3 works round an oddity of the Raspberry Pi’s CPU, discussed in more detail in the original blog post. We need to ensure that QEMU only runs on this isolated CPU. In the original post this was done by patching QEMU but there is a much easier way, taskset. Taskset allows us to restrict QEMU to only CPU 3 with the “-c 3-3” option.
This is my basic run script to boot an ARM VM on the Raspberry Pi2. I used the Linaro prebuilt ARM kernels and root images to test with and have included the URLs in the script.
#!/bin/sh # Disable the QEMU sound driver export QEMU_AUDIO_DRV=none # Basic system setup an ARM vexpress with 1 CPU, 256M of RAM smp=1 cpu=host ram=256 machine=vexpress-a15 # Where are the kernel and root images stored dir=/root/linaro # Source: https://snapshots.linaro.org/ubuntu/images/kvm/latest/zImage-armv7 kernel=$dir/zImage-armv7 # Source: https://snapshots.linaro.org/ubuntu/images/kvm/latest/vexpress-v2p-ca15-tc1.dtb dtb=$dir/vexpress-v2p-ca15-tc1.dtb # Source: http://snapshots.linaro.org/ubuntu/images/kvm-guest/36/armhf/kvm-armhf.qcow2.xz rootfs=$dir/kvm-armhf.qcow2 # Virtual machine Linux command line cmdline="root=/dev/vda2 rw mem=${ram}M console=ttyAMA0 rootwait rootfstype=ext4" # Use taskset to ensure that QEMU only runs on cpu 3 taskset -c 3-3 qemu-system-arm -enable-kvm -smp $smp -m $ram -M $machine -cpu host -kernel $kernel -dtb $dtb -append "$cmdline" -drive if=none,id=rootfs,file=$rootfs -device virtio-blk-device,drive=rootfs -netdev user,id=user0 -device virtio-net-device,netdev=user0 -nographic