KVM on the Raspberry Pi2

In my last post I wrote about to getting my Pi2 to boot with HYP enabled on all 4 CPUs. The next stage is to get a kernel with KVM enabled and get a VM up and running. Once again most of this method is taken from a blog post by I have tidied it up and refined the method of using a single CPU core without patching QEMU.

Building a KVM enabled kernel for the Pi

First of all you need to get a KVM enabled kernel for the Pi2 host. These commands checkout the current 3.18 version of the Raspberry Pi foundation’s kernel tree and apply a pull request on top of it to enable GIC emulation.

git clone https://github.com/raspberrypi/linux
cd linux
git checkout rpi-3.18.y
git fetch origin pull/902/head:VGIC-emu
git checkout VGIC-emu

I started off with the same base config as the kernel on my Pi 2. There is a file /proc/config.gz that stores the config the running kernel was built with. Copy this file to your build host, uncompress it and rename it to /working/rpi2/.config that way you will only need to make a few minor changes to the config.

scp user@pi:/proc/config.gz /tmp/config.gz
mkdir -p /working/rpi2
zcat /tmp/config.gz > /working/rpi2/.config

Now we need to setup a cross compile environment again and start the kernel config system.

export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-
make O=/working/rpi2 menuconfig

I’ve listed the options that need to be changed to enable KVM support in the kernel, these notes are copied from ‘s post. I have uploaded my KVM enabled config with these options set already.

Note the O= option this sets the directory where compiled parts of the kernel are put, it means that your source tree stay clean.

  • Patch physical to virtual translations at runtime
  • General setup -> Control Group support
  • System Type -> Support for Large Physical Address Extension
  • Boot options -> Use appended device tree blob to zImage (EXPERIMENTAL)
  • Boot options -> Supplement the appended DTB with traditional ATAG information
  • Virtualization -> Kernel-based Virtual Machine (KVM) support (NEW)
    • DISABLE Virtualization -> KVM support for Virtual GIC
    • ENABLE Virtualization -> KVM support for Emulated GIC

Now you just need to build the kernel and install the modules

make O=/working/rpi2 all -j 4
export INSTALL_MOD_PATH=/tmp/rpi-kernel
make O=/working/rpi2 modules_install

To build the Pi2 boot image, glue the kernel and DTB file together thus

cat /working/rpi2/arch/arm/boot/zImage /working/rpi2/arch/arm/boot/dts/bcm2709-rpi-2-b.dtb > /tmp/rpi-kernel/kernel7.img

Copy the directory /tmp/rpi-kernel to the Pi2, replace /boot/firmware/kernel7.img (take a backup first) with our new kernel and move the new 3.18.x modules directory to /lib/modules/.

You will need to adjust the Pi’s kernel command line to add “isolcpus=3” to work round a bug. To do this via u-boot run these commands from the u-boot command line:

setenv "${bootargs} isolcpus=3"

A final reboot and you should get this in dmesg

dmesg | grep kvm
kvm [1]: timer IRQ99
kvm [1]: Hyp mode initialized successfully

Booting your first Virtual Machine

So long as you are running Debian / Raspbian Jessie then you can just run

apt-get install qemu-system

Adding the boot option isolcpus=3 works round an oddity of the Raspberry Pi’s CPU, discussed in more detail in the original blog post. We need to ensure that QEMU only runs on this isolated CPU. In the original post this was done by patching QEMU but there is a much easier way, taskset. Taskset allows us to restrict QEMU to only CPU 3 with the “-c 3-3” option.

This is my basic run script to boot an ARM VM on the Raspberry Pi2. I used the Linaro prebuilt ARM kernels and root images to test with and have included the URLs in the script.


# Disable the QEMU sound driver
export QEMU_AUDIO_DRV=none

# Basic system setup an ARM vexpress with 1 CPU, 256M of RAM
# Where are the kernel and root images stored

# Source: https://snapshots.linaro.org/ubuntu/images/kvm/latest/zImage-armv7 
# Source: https://snapshots.linaro.org/ubuntu/images/kvm/latest/vexpress-v2p-ca15-tc1.dtb 
# Source: http://snapshots.linaro.org/ubuntu/images/kvm-guest/36/armhf/kvm-armhf.qcow2.xz
# Virtual machine Linux command line
cmdline="root=/dev/vda2 rw mem=${ram}M console=ttyAMA0 rootwait rootfstype=ext4"

# Use taskset to ensure that QEMU only runs on cpu 3
taskset -c 3-3 qemu-system-arm -enable-kvm -smp $smp -m $ram -M $machine -cpu host -kernel $kernel -dtb $dtb -append "$cmdline" -drive if=none,id=rootfs,file=$rootfs -device virtio-blk-device,drive=rootfs -netdev user,id=user0 -device virtio-net-device,netdev=user0 -nographic


Booting a Raspberry Pi2, with u-boot and HYP enabled

I have been playing with my Raspberry Pi2 and a nearly pure build of Debian Jessie. I have now got u-boot and QEMU with hardware acceleration (kvm) working in a fairly clean way.

There are a lot of parts to getting this all working correctly, and I have done this by gluing together a number of blog posts, mailing list posts and a bit of arm knowledge.

First thing to do is to get a working cross compile environment, so you can build on modern x86 hardware which is a lot fast then building on the Pi. Fortunately in modern Debian or Ubuntu it’s as simple as these commands:

sudo apt-get install gcc-arm-linux-gnueabihf
export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-

Booting a Raspberry Pi 2 with u-boot

Using u-boot isn’t strictly needed but I much prefer u-boot to the Pi’s normal raw kernel boot. Mainline u-boot has support for the Raspberry Pi 2 so it’s a fairly simple process of:

git clone git://git.denx.de/u-boot.git
cd u-boot
make rpi_2_defconfig
make all

This will give you a u-boot.bin binary that will work on the Raspberry Pi2, transfer this to your Raspberry Pi and change the kernel in config.txt to read:


For the next part you will need a serial console on your Raspberry Pi because the Pi will not autoboot this time. Reboot your Pi and you should see the following message on your serial console:

U-Boot 2015.04-00631-gace97d2 (May 03 2015 - 10:52:52)

DRAM:  944 MiB
WARNING: Caches not enabled
RPI: Board rev 16 outside known range
RPI Unknown model
MMC:   bcm2835_sdhci: 0

In:    serial
Out:   lcd
Err:   lcd
Net:   Net Initialization Skipped
No ethernet found.
Hit any key to stop autoboot:  0

I suggest that you “hand boot” the Pi the first time, to make sure everything works.

# Tell Linux that it is booting on a Raspberry Pi2
setenv machid 0x00000c42
# Set the kernel boot command line
setenv bootargs "earlyprintk console=tty0 console=ttyAMA0 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait noinitrd"
# Save these changes to u-boot's environment
# Load the existing Linux kernel into RAM
fatload mmc 0:1 ${kernel_addr_r} kernel7.img
# Boot the kernel we have just loaded
bootz ${kernel_addr_r}

You should now see Linux booting, I have a little boot script setup to run the last 2 commands automatically.  Put the commands in file and use mkimage to build a u-boot wrapper round them. Note this example assumes that you are using the Debian Jessie image I linked at the top, if you are using Raspbian then you will need to use /boot/boot.scr as the output path.

mkimage -A arm -O linux -T script -C none -a 0x00000000 -e 0x00000000 -n "RPi2 Boot Script" -d /path/to/script /boot/firmware/boot.scr

U-Boot and HYP mode

The next stage is to allow the Raspberry Pi2 to boot with HYP mode enabled on the CPU. This requires starting in secure mode, enabling HYP and then jumping to u-boot or Linux. I didn’t work this myself I lifted the code and the technique from a blog post by a NetBSD developer. He does a much better job of explaining the how and why of this if you are interested.

To build the bootloader stub, clone the git https://github.com/slp/rpi2-hyp-boot you will need to fix the the gcc path or use the fixed version in my Github repo. To build type make and you should end up with a file bootblk.bin. The build process is so fast you can build it on the Pi without a problem.

Run this command to stick the HYP bootblock on front of u-boot.bin, again this assumes you are running the Debian Jessie image at the top this post.

cat bootblk.bin /boot/firmware/u-boot.bin > /boot/firmware/u-boot.hyp

You will also need to change /boot/firmware/config.txt to contain the kernel_old option so that the GPU bootloader boots the kernel with secure mode enabled. The contents should now be:


Reboot the Pi again and you should see u-boot and then Linux boot as normal, login and run this command and you should see: “CPU: All CPU(s) started in HYP mode.”

dmesg | grep "All CPU"

My next post will be about enabling KVM on the Pi2 and booting your first VM.

Masterclass 2015 hardware challenge

Last week I competed in the UK Cyber Security Challenge, spending 2 days in London on the HMS Belfast. I was fortunate to be part of a great team, including the event’s overall winner Adam Tonks.

Part of the challenge was a hardware badge, we were given “suspicious” items that had been intercepted and asked to find out what they are.2015-03-20 22.10.58Other than the USB port and central chip, the next thing I noticed was the five gold pads in the middle of the board. Because I tinker with embedded systems, I thought they look a lot like a serial debug port. In fact I don’t think they are are a serial port any more but serial port is a good clue. There is a row of 8 LEDs labelled LED1-8 which become important at the end of the puzzle.

When you plug the device in a slew of USB entries in appear in dmesg, the key lines are in bold. The first line tells you the USB device is made by freescale who make ARM chips which are tiny computers. The next line about ttyACM0 and attaching the removable scsi disk we’ll use to explore the badge in more detail.

usb 1-4.1.2: new full-speed USB device number 39 using ehci-pci
usb 1-4.1.2: New USB device found, idVendor=15a2, idProduct=0800
usb 1-4.1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
usb 1-4.1.2: Product: MSD_CDC DEVICE
usb 1-4.1.2: Manufacturer: FREESCALE SEMICONDUCTOR INC.
cdc_acm 1-4.1.2:1.0: This device cannot do calls on its own. It is not a modem.
cdc_acm 1-4.1.2:1.0: ttyACM0: USB ACM device
usb-storage 1-4.1.2:1.2: USB Mass Storage device detected
scsi20 : usb-storage 1-4.1.2:1.2
scsi 20:0:0:0: Direct-Access FSL SEMI FSL MASS STORAGE 0001 PQ: 0 ANSI: 4
sd 20:0:0:0: Attached scsi generic sg6 type 0
sd 20:0:0:0: [sdf] 8 512-byte logical blocks: (4.09 kB/4.00 KiB)
sd 20:0:0:0: [sdf] Write Protect is off
sd 20:0:0:0: [sdf] Mode Sense: 00 00 00 00
sd 20:0:0:0: [sdf] Asking for cache data failed
sd 20:0:0:0: [sdf] Assuming drive cache: write through
sd 20:0:0:0: [sdf] Asking for cache data failed
sd 20:0:0:0: [sdf] Assuming drive cache: write through
Dev sdf: unable to read RDB block 8
sdf: unable to read partition table
sdf: partition table beyond EOD, truncated
sd 20:0:0:0: [sdf] Attached SCSI removable disk

First question what is on the USB serial port?

I normally use screen to access serial ports when I am working with embedded systems, it’s easier to use than minicom and putting the -L flag writes a log of the serial session. This is important for forensic and incident response work as it records your artifacts. The number at the end is the serial port speed, however the badge is very forgiving of different speeds, other embedded devices are much less so.

screen -L /dev/ttyACM0 115200

Press return, and we get a password prompt but what’s the password?

Sorry, wrong password.

We tried various passwords related to our previous investigations but not of them worked.

The next thought was to check the USB mass storage device, the device is tiny only 4k in size. Mostly the device is just empty space and doesn’t have a filesystem on it so mounting it doesn’t help. In the heat of the challenge I used strings to “eyeball” the raw contents of the device, but the output of xxd is neater. In the raw dump we found some interesting data.

sudo xxd -a /dev/sdf
0000000: 0000 5365 6375 7265 2070 6173 7377 6f72  ..Secure passwor
0000010: 6420 6469 736b 0000 0000 0000 0000 0000  d disk..........
0000020: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000200: 0053 6563 7572 6520 7061 7373 776f 7264  .Secure password
0000210: 2073 746f 7261 6765 2064 6174 6162 6173   storage databas
0000220: 650d 6164 6d69 6e3a 6e69 6d64 610d 6f70  e.admin:nimda.op
0000230: 6572 6174 6f72 3a62 336c 6661 3574 3233  erator:b3lfa5t23
0000240: 0d00 0000 0000 0000 0000 0000 0000 0000  ................
0000250: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000400: 0053 5550 4552 2053 6563 7572 6520 7061  .SUPER Secure pa
0000410: 7373 776f 7264 2073 746f 7261 6765 2064  ssword storage d
0000420: 6174 6162 6173 650d 0061 6d68 3564 4456  atabase..amh5dDV
0000430: 305a 6d51 3d00 0000 0000 0000 0000 0000  0ZmQ=...........
0000440: 0000 0000 0000 0000 0000 0000 0000 0000  ................
0000ff0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

A few plain text passwords, and then what looks to be the inevitable base64 encoded string in the “SUPER Secure password storage database”. Simply use base64 -d to decode the string and we get what could be a password.

echo amh5dDV0ZmQ= | base64 -d

Using this string as password allows access to a menu, option 1 doesn’t work and option 2 needs a password.

Welcome to your new secure badge token. Please start initialisation.
     1 - - - - - Network test
     2 - - - - - Superuser mode
     3 - - - - - Logout
Sorry, this device does not support this.
     1 - - - - - Network test
     2 - - - - - Superuser mode
     3 - - - - - Logout
Entering enable mode...
Enable password:*

Again try a few passwords without success, Adam worked out that you can buffer overflow the menu and get a crash dump from the menu process.

Entering enable mode...
Enable password:***********************************************************************************************************************
main_thread(16788): Oops!
 CPU[1]: local_irq_count[0] irqs_running[1] 
 memory DUMP starting 0x004005 
 01 7F 8E 9E 8D D0 55 E3 
64 62 67 6E 77 00 78 66
6D 41 49 6C 20 6D 65 21
6A 6F 65 6C 40 62 2E 63 
 6F 6D FF 00 FF 00 FF 00

Another leap of Adam’s led to translating the hex dump into binary using perl, and decoding the resulting binary with xxd. This gave more text, an easter egg left by the designer of the puzzle Joel from BT and a null terminated string of 5 characters, lets see if they are a password. I’ve highlighted in bold this string in the various outputs.

perl -E'print pack "H*","017F8E9E8DD055E36462676E770078666D41496C206D65216A6F656C40622E636F6DFF00FF00FF00"'| xxd

0000000: 017f 8e9e 8dd0 55e3 6462 676e 7700 7866 ......U.dbgnw.xf
0000010: 6d41 496c 206d 6521 6a6f 656c 4062 2e63 mAIl me!joel@b.c
0000020: 6f6d ff00 ff00 ff00                     om......

The string found in the crash dump does indeed allow access to the final menu that lets find the badge’s key.

Welcome to your new secure badge token. Please start initialisation.
     1 - - - - - Network test
     2 - - - - - Superuser mode
     3 - - - - - Logout
Entering enable mode...
Enable password:**
     1 - - - - - Debug optical network
     2 - - - - - Logout

How the badge sends the key is the most fun part I think, when you debug the optical network the badge flashes the 8 LEDs on the side of the badge. Again Adam spotted that if you transcribe them as binary the first 2 characters are PW and then 4 random characters which is the badge “ultimate” password.

     1 - - - - - Debug optical network
     2 - - - - - Logout
transmitting key...
Format: PW<key>

     1 - - - - - Debug optical network
     2 - - - - - Logout

I’ve uploaded a video of my badge transmitting it’s key, see if you can decode it.

Mounting Android encrypted storage on the Galaxy S2

Because I like to understand how things work before they break, I wanted to be able to mount the encrypted storage of my Galaxy S2 phone from the ADB shell.

My understanding of the Android encryption system is that there is a metadata file stored in /efs at least for phones that have an efs partition. This metadata file contains the normal LUKS headers and encryption keys.

The following works for me in the ADB shell, it mounts the efs partition and then requests that vold unlocks the storage.

mkdir /efs
mount /dev/block/mmcblk0p1 /efs
setprop ro.crypto.state encrypted
vdc cryptfs checkpw 'your passphrase here'
mount -o ro /dev/block/dm-0 /data