Fixing LDAP error 53 “Server is unwilling to perform”

Work has a pair of OpenLDAP servers which are in a standard master / slave synchronized setup. While preparing for some updates I checked that the LDAP servers where syncing correctly and discovered that the slave hadn’t updated in over 6 months!

On the slave server /var/log/syslog contained the following errors:

slapd[PID]: do_syncrep2: rid=XXX LDAP_RES_SEARCH_RESULT (53) Server is unwilling to perform
slapd[PID]: do_syncrep2: rid=XXX (53) Server is unwilling to perform

Working through the Ubuntu server guide used to set up the pair of servers in the first place didn’t shed any light on the problem. A fresh Ubuntu 14.04 server in GCE showed the same problem, so at least I know the problem is on the master server.

I finally got a clue from chapter 12 of “LDAP for Rocket Scientists“, which suggested that the master server had “no global superior knowledge”. This was enough to make me test removing the accesslog databases, which track LDAP transactions and allow slave servers to sync changes from the master.

In the end it was a simple as removing the databases from /var/lib/ldap/accesslog and letting slapd rebuild them after a restart. Note depending on the config in slapd this might be in a different directory, check the setting for olcDbDirectory with this command:

sudo slapcat -b cn=config -a "(|(cn=config)(olcDatabase={2}hdb))"

Exchange Cumulative Update 8 and SSL certificates

This weekend while I was patching and rebooting KVM systems for Venom I took the opportunity to apply Microsoft’s latest Exchange Cumulative Update 8 work’s Exchange server.

I ran the pre-upgrade checks and  they picked up that my user wasn’t in the correct AD groups for scheme updates, once that was fixed the upgrade started without problems.

When the upgrade got to Section 10 – Mailbox role: Transport Services the upgrade failed because a certificate had expired. Fine just install an updated certificate but all of the Exchange management tools have been uninstalled so you can’t get to the certificate.

When you rerun the installer it detects the failed install and tries to resume and fails at the same place, in the end I had to move the clock back two days on the server to get the management tools to install so that the certificate could be replaced.

I’ve reviewed the Microsoft documentation and I can’t see any reference to this problem and it wasn’t detected in the pre-upgrade checks. I’m sure there is some powershell magic that could fix this but at 1am on a Monday morning I wasn’t all that interested in finding out!

KVM on the Raspberry Pi2

In my last post I wrote about to getting my Pi2 to boot with HYP enabled on all 4 CPUs. The next stage is to get a kernel with KVM enabled and get a VM up and running. Once again most of this method is taken from a blog post by I have tidied it up and refined the method of using a single CPU core without patching QEMU.

Building a KVM enabled kernel for the Pi

First of all you need to get a KVM enabled kernel for the Pi2 host. These commands checkout the current 3.18 version of the Raspberry Pi foundation’s kernel tree and apply a pull request on top of it to enable GIC emulation.

git clone https://github.com/raspberrypi/linux
cd linux
git checkout rpi-3.18.y
git fetch origin pull/902/head:VGIC-emu
git checkout VGIC-emu

I started off with the same base config as the kernel on my Pi 2. There is a file /proc/config.gz that stores the config the running kernel was built with. Copy this file to your build host, uncompress it and rename it to /working/rpi2/.config that way you will only need to make a few minor changes to the config.

scp user@pi:/proc/config.gz /tmp/config.gz
mkdir -p /working/rpi2
zcat /tmp/config.gz > /working/rpi2/.config

Now we need to setup a cross compile environment again and start the kernel config system.

export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-
make O=/working/rpi2 menuconfig

I’ve listed the options that need to be changed to enable KVM support in the kernel, these notes are copied from ‘s post. I have uploaded my KVM enabled config with these options set already.

Note the O= option this sets the directory where compiled parts of the kernel are put, it means that your source tree stay clean.

  • Patch physical to virtual translations at runtime
  • General setup -> Control Group support
  • System Type -> Support for Large Physical Address Extension
  • Boot options -> Use appended device tree blob to zImage (EXPERIMENTAL)
  • Boot options -> Supplement the appended DTB with traditional ATAG information
  • Virtualization -> Kernel-based Virtual Machine (KVM) support (NEW)
    • DISABLE Virtualization -> KVM support for Virtual GIC
    • ENABLE Virtualization -> KVM support for Emulated GIC

Now you just need to build the kernel and install the modules

make O=/working/rpi2 all -j 4
export INSTALL_MOD_PATH=/tmp/rpi-kernel
make O=/working/rpi2 modules_install

To build the Pi2 boot image, glue the kernel and DTB file together thus

cat /working/rpi2/arch/arm/boot/zImage /working/rpi2/arch/arm/boot/dts/bcm2709-rpi-2-b.dtb > /tmp/rpi-kernel/kernel7.img

Copy the directory /tmp/rpi-kernel to the Pi2, replace /boot/firmware/kernel7.img (take a backup first) with our new kernel and move the new 3.18.x modules directory to /lib/modules/.

You will need to adjust the Pi’s kernel command line to add “isolcpus=3” to work round a bug. To do this via u-boot run these commands from the u-boot command line:

setenv "${bootargs} isolcpus=3"
saveenv

A final reboot and you should get this in dmesg

dmesg | grep kvm
kvm [1]: timer IRQ99
kvm [1]: Hyp mode initialized successfully

Booting your first Virtual Machine

So long as you are running Debian / Raspbian Jessie then you can just run

apt-get install qemu-system

Adding the boot option isolcpus=3 works round an oddity of the Raspberry Pi’s CPU, discussed in more detail in the original blog post. We need to ensure that QEMU only runs on this isolated CPU. In the original post this was done by patching QEMU but there is a much easier way, taskset. Taskset allows us to restrict QEMU to only CPU 3 with the “-c 3-3” option.

This is my basic run script to boot an ARM VM on the Raspberry Pi2. I used the Linaro prebuilt ARM kernels and root images to test with and have included the URLs in the script.

#!/bin/sh

# Disable the QEMU sound driver
export QEMU_AUDIO_DRV=none

# Basic system setup an ARM vexpress with 1 CPU, 256M of RAM
smp=1
cpu=host
ram=256
machine=vexpress-a15
# Where are the kernel and root images stored
dir=/root/linaro

# Source: https://snapshots.linaro.org/ubuntu/images/kvm/latest/zImage-armv7 
kernel=$dir/zImage-armv7
# Source: https://snapshots.linaro.org/ubuntu/images/kvm/latest/vexpress-v2p-ca15-tc1.dtb 
dtb=$dir/vexpress-v2p-ca15-tc1.dtb
# Source: http://snapshots.linaro.org/ubuntu/images/kvm-guest/36/armhf/kvm-armhf.qcow2.xz
rootfs=$dir/kvm-armhf.qcow2
# Virtual machine Linux command line
cmdline="root=/dev/vda2 rw mem=${ram}M console=ttyAMA0 rootwait rootfstype=ext4"

# Use taskset to ensure that QEMU only runs on cpu 3
taskset -c 3-3 qemu-system-arm -enable-kvm -smp $smp -m $ram -M $machine -cpu host -kernel $kernel -dtb $dtb -append "$cmdline" -drive if=none,id=rootfs,file=$rootfs -device virtio-blk-device,drive=rootfs -netdev user,id=user0 -device virtio-net-device,netdev=user0 -nographic

 

Booting a Raspberry Pi2, with u-boot and HYP enabled

I have been playing with my Raspberry Pi2 and a nearly pure build of Debian Jessie. I have now got u-boot and QEMU with hardware acceleration (kvm) working in a fairly clean way.

There are a lot of parts to getting this all working correctly, and I have done this by gluing together a number of blog posts, mailing list posts and a bit of arm knowledge.

First thing to do is to get a working cross compile environment, so you can build on modern x86 hardware which is a lot fast then building on the Pi. Fortunately in modern Debian or Ubuntu it’s as simple as these commands:

sudo apt-get install gcc-arm-linux-gnueabihf
export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-

Booting a Raspberry Pi 2 with u-boot

Using u-boot isn’t strictly needed but I much prefer u-boot to the Pi’s normal raw kernel boot. Mainline u-boot has support for the Raspberry Pi 2 so it’s a fairly simple process of:

git clone git://git.denx.de/u-boot.git
cd u-boot
make rpi_2_defconfig
make all

This will give you a u-boot.bin binary that will work on the Raspberry Pi2, transfer this to your Raspberry Pi and change the kernel in config.txt to read:

kernel=u-boot.bin

For the next part you will need a serial console on your Raspberry Pi because the Pi will not autoboot this time. Reboot your Pi and you should see the following message on your serial console:

U-Boot 2015.04-00631-gace97d2 (May 03 2015 - 10:52:52)

DRAM:  944 MiB
WARNING: Caches not enabled
RPI: Board rev 16 outside known range
RPI Unknown model
MMC:   bcm2835_sdhci: 0

In:    serial
Out:   lcd
Err:   lcd
Net:   Net Initialization Skipped
No ethernet found.
Hit any key to stop autoboot:  0

I suggest that you “hand boot” the Pi the first time, to make sure everything works.

# Tell Linux that it is booting on a Raspberry Pi2
setenv machid 0x00000c42
# Set the kernel boot command line
setenv bootargs "earlyprintk console=tty0 console=ttyAMA0 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait noinitrd"
# Save these changes to u-boot's environment
saveenv
# Load the existing Linux kernel into RAM
fatload mmc 0:1 ${kernel_addr_r} kernel7.img
# Boot the kernel we have just loaded
bootz ${kernel_addr_r}

You should now see Linux booting, I have a little boot script setup to run the last 2 commands automatically.  Put the commands in file and use mkimage to build a u-boot wrapper round them. Note this example assumes that you are using the Debian Jessie image I linked at the top, if you are using Raspbian then you will need to use /boot/boot.scr as the output path.

mkimage -A arm -O linux -T script -C none -a 0x00000000 -e 0x00000000 -n "RPi2 Boot Script" -d /path/to/script /boot/firmware/boot.scr

U-Boot and HYP mode

The next stage is to allow the Raspberry Pi2 to boot with HYP mode enabled on the CPU. This requires starting in secure mode, enabling HYP and then jumping to u-boot or Linux. I didn’t work this myself I lifted the code and the technique from a blog post by a NetBSD developer. He does a much better job of explaining the how and why of this if you are interested.

To build the bootloader stub, clone the git https://github.com/slp/rpi2-hyp-boot you will need to fix the the gcc path or use the fixed version in my Github repo. To build type make and you should end up with a file bootblk.bin. The build process is so fast you can build it on the Pi without a problem.

Run this command to stick the HYP bootblock on front of u-boot.bin, again this assumes you are running the Debian Jessie image at the top this post.

cat bootblk.bin /boot/firmware/u-boot.bin > /boot/firmware/u-boot.hyp

You will also need to change /boot/firmware/config.txt to contain the kernel_old option so that the GPU bootloader boots the kernel with secure mode enabled. The contents should now be:

kernel=u-boot.hyp
kernel_old=1

Reboot the Pi again and you should see u-boot and then Linux boot as normal, login and run this command and you should see: “CPU: All CPU(s) started in HYP mode.”

dmesg | grep "All CPU"

My next post will be about enabling KVM on the Pi2 and booting your first VM.