Fixing LDAP error 53 “Server is unwilling to perform”

Work has a pair of OpenLDAP servers which are in a standard master / slave synchronized setup. While preparing for some updates I checked that the LDAP servers where syncing correctly and discovered that the slave hadn’t updated in over 6 months!

On the slave server /var/log/syslog contained the following errors:

slapd[PID]: do_syncrep2: rid=XXX LDAP_RES_SEARCH_RESULT (53) Server is unwilling to perform
slapd[PID]: do_syncrep2: rid=XXX (53) Server is unwilling to perform

Working through the Ubuntu server guide used to set up the pair of servers in the first place didn’t shed any light on the problem. A fresh Ubuntu 14.04 server in GCE showed the same problem, so at least I know the problem is on the master server.

I finally got a clue from chapter 12 of “LDAP for Rocket Scientists“, which suggested that the master server had “no global superior knowledge”. This was enough to make me test removing the accesslog databases, which track LDAP transactions and allow slave servers to sync changes from the master.

In the end it was a simple as removing the databases from /var/lib/ldap/accesslog and letting slapd rebuild them after a restart. Note depending on the config in slapd this might be in a different directory, check the setting for olcDbDirectory with this command:

sudo slapcat -b cn=config -a "(|(cn=config)(olcDatabase={2}hdb))"

KVM on the Raspberry Pi2

In my last post I wrote about to getting my Pi2 to boot with HYP enabled on all 4 CPUs. The next stage is to get a kernel with KVM enabled and get a VM up and running. Once again most of this method is taken from a blog post by I have tidied it up and refined the method of using a single CPU core without patching QEMU.

Building a KVM enabled kernel for the Pi

First of all you need to get a KVM enabled kernel for the Pi2 host. These commands checkout the current 3.18 version of the Raspberry Pi foundation’s kernel tree and apply a pull request on top of it to enable GIC emulation.

git clone
cd linux
git checkout rpi-3.18.y
git fetch origin pull/902/head:VGIC-emu
git checkout VGIC-emu

I started off with the same base config as the kernel on my Pi 2. There is a file /proc/config.gz that stores the config the running kernel was built with. Copy this file to your build host, uncompress it and rename it to /working/rpi2/.config that way you will only need to make a few minor changes to the config.

scp user@pi:/proc/config.gz /tmp/config.gz
mkdir -p /working/rpi2
zcat /tmp/config.gz > /working/rpi2/.config

Now we need to setup a cross compile environment again and start the kernel config system.

export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-
make O=/working/rpi2 menuconfig

I’ve listed the options that need to be changed to enable KVM support in the kernel, these notes are copied from ‘s post. I have uploaded my KVM enabled config with these options set already.

Note the O= option this sets the directory where compiled parts of the kernel are put, it means that your source tree stay clean.

  • Patch physical to virtual translations at runtime
  • General setup -> Control Group support
  • System Type -> Support for Large Physical Address Extension
  • Boot options -> Use appended device tree blob to zImage (EXPERIMENTAL)
  • Boot options -> Supplement the appended DTB with traditional ATAG information
  • Virtualization -> Kernel-based Virtual Machine (KVM) support (NEW)
    • DISABLE Virtualization -> KVM support for Virtual GIC
    • ENABLE Virtualization -> KVM support for Emulated GIC

Now you just need to build the kernel and install the modules

make O=/working/rpi2 all -j 4
export INSTALL_MOD_PATH=/tmp/rpi-kernel
make O=/working/rpi2 modules_install

To build the Pi2 boot image, glue the kernel and DTB file together thus

cat /working/rpi2/arch/arm/boot/zImage /working/rpi2/arch/arm/boot/dts/bcm2709-rpi-2-b.dtb > /tmp/rpi-kernel/kernel7.img

Copy the directory /tmp/rpi-kernel to the Pi2, replace /boot/firmware/kernel7.img (take a backup first) with our new kernel and move the new 3.18.x modules directory to /lib/modules/.

You will need to adjust the Pi’s kernel command line to add “isolcpus=3” to work round a bug. To do this via u-boot run these commands from the u-boot command line:

setenv "${bootargs} isolcpus=3"

A final reboot and you should get this in dmesg

dmesg | grep kvm
kvm [1]: timer IRQ99
kvm [1]: Hyp mode initialized successfully

Booting your first Virtual Machine

So long as you are running Debian / Raspbian Jessie then you can just run

apt-get install qemu-system

Adding the boot option isolcpus=3 works round an oddity of the Raspberry Pi’s CPU, discussed in more detail in the original blog post. We need to ensure that QEMU only runs on this isolated CPU. In the original post this was done by patching QEMU but there is a much easier way, taskset. Taskset allows us to restrict QEMU to only CPU 3 with the “-c 3-3” option.

This is my basic run script to boot an ARM VM on the Raspberry Pi2. I used the Linaro prebuilt ARM kernels and root images to test with and have included the URLs in the script.


# Disable the QEMU sound driver
export QEMU_AUDIO_DRV=none

# Basic system setup an ARM vexpress with 1 CPU, 256M of RAM
# Where are the kernel and root images stored

# Source: 
# Source: 
# Source:
# Virtual machine Linux command line
cmdline="root=/dev/vda2 rw mem=${ram}M console=ttyAMA0 rootwait rootfstype=ext4"

# Use taskset to ensure that QEMU only runs on cpu 3
taskset -c 3-3 qemu-system-arm -enable-kvm -smp $smp -m $ram -M $machine -cpu host -kernel $kernel -dtb $dtb -append "$cmdline" -drive if=none,id=rootfs,file=$rootfs -device virtio-blk-device,drive=rootfs -netdev user,id=user0 -device virtio-net-device,netdev=user0 -nographic


Booting a Raspberry Pi2, with u-boot and HYP enabled

I have been playing with my Raspberry Pi2 and a nearly pure build of Debian Jessie. I have now got u-boot and QEMU with hardware acceleration (kvm) working in a fairly clean way.

There are a lot of parts to getting this all working correctly, and I have done this by gluing together a number of blog posts, mailing list posts and a bit of arm knowledge.

First thing to do is to get a working cross compile environment, so you can build on modern x86 hardware which is a lot fast then building on the Pi. Fortunately in modern Debian or Ubuntu it’s as simple as these commands:

sudo apt-get install gcc-arm-linux-gnueabihf
export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-

Booting a Raspberry Pi 2 with u-boot

Using u-boot isn’t strictly needed but I much prefer u-boot to the Pi’s normal raw kernel boot. Mainline u-boot has support for the Raspberry Pi 2 so it’s a fairly simple process of:

git clone git://
cd u-boot
make rpi_2_defconfig
make all

This will give you a u-boot.bin binary that will work on the Raspberry Pi2, transfer this to your Raspberry Pi and change the kernel in config.txt to read:


For the next part you will need a serial console on your Raspberry Pi because the Pi will not autoboot this time. Reboot your Pi and you should see the following message on your serial console:

U-Boot 2015.04-00631-gace97d2 (May 03 2015 - 10:52:52)

DRAM:  944 MiB
WARNING: Caches not enabled
RPI: Board rev 16 outside known range
RPI Unknown model
MMC:   bcm2835_sdhci: 0

In:    serial
Out:   lcd
Err:   lcd
Net:   Net Initialization Skipped
No ethernet found.
Hit any key to stop autoboot:  0

I suggest that you “hand boot” the Pi the first time, to make sure everything works.

# Tell Linux that it is booting on a Raspberry Pi2
setenv machid 0x00000c42
# Set the kernel boot command line
setenv bootargs "earlyprintk console=tty0 console=ttyAMA0 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait noinitrd"
# Save these changes to u-boot's environment
# Load the existing Linux kernel into RAM
fatload mmc 0:1 ${kernel_addr_r} kernel7.img
# Boot the kernel we have just loaded
bootz ${kernel_addr_r}

You should now see Linux booting, I have a little boot script setup to run the last 2 commands automatically.  Put the commands in file and use mkimage to build a u-boot wrapper round them. Note this example assumes that you are using the Debian Jessie image I linked at the top, if you are using Raspbian then you will need to use /boot/boot.scr as the output path.

mkimage -A arm -O linux -T script -C none -a 0x00000000 -e 0x00000000 -n "RPi2 Boot Script" -d /path/to/script /boot/firmware/boot.scr

U-Boot and HYP mode

The next stage is to allow the Raspberry Pi2 to boot with HYP mode enabled on the CPU. This requires starting in secure mode, enabling HYP and then jumping to u-boot or Linux. I didn’t work this myself I lifted the code and the technique from a blog post by a NetBSD developer. He does a much better job of explaining the how and why of this if you are interested.

To build the bootloader stub, clone the git you will need to fix the the gcc path or use the fixed version in my Github repo. To build type make and you should end up with a file bootblk.bin. The build process is so fast you can build it on the Pi without a problem.

Run this command to stick the HYP bootblock on front of u-boot.bin, again this assumes you are running the Debian Jessie image at the top this post.

cat bootblk.bin /boot/firmware/u-boot.bin > /boot/firmware/u-boot.hyp

You will also need to change /boot/firmware/config.txt to contain the kernel_old option so that the GPU bootloader boots the kernel with secure mode enabled. The contents should now be:


Reboot the Pi again and you should see u-boot and then Linux boot as normal, login and run this command and you should see: “CPU: All CPU(s) started in HYP mode.”

dmesg | grep "All CPU"

My next post will be about enabling KVM on the Pi2 and booting your first VM.

Linux Capabilities and rsync, from presentation to practice

Hazel Smith gave an excellent talk at FLOSSUK’s Unconference in London about Linux Capabilities and using them as part of “least privilege” when running backups of Linux systems.

Hazel explained that by using Capabilities you can allow a single user (in this example backuphelper) when running a single binary (rsync) to read any file on the system. Hazel stressed in the presentation that this isn’t a privilege to give out lightly, however the ability to read any file isn’t a direct path to root. This level of privilege will allow access to for example hashed passwords or contents of any user’s files.

My personal backups use BackupPC which can pull backups with tar, smbclient or as I use rsync over ssh.

The second half of this post documents how I put Hazel’s talk into practice on my Debian based systems. I have also uploaded to GitHub an example Ansible task I used to roll out the changes to my systems.

Target System Setup

First off install the support packages for capabilities.

sudo apt-get install libcap2-bin libpam-cap
sudo pam-auth-update

Run  pam-auth-update and enable “Inheritable Capabilities Management”, if you prefer to manually manage the pam config files then add the line “auth optional” to /etc/pam.d/common-auth

You will also need to add following line to /etc/security/capability.conf to allow backuphelper to retain cap_dac_read_search. The rules applied in order so make sure it’s above the default deny line “none  *”

cap_dac_read_search backuphelper

This next command sets cap_dac_read_search as Inheritable and Effective for the rsync binary. The net effect is that when the backuphelper user runs rsync that process can read any file on the system.

sudo setcap cap_dac_read_search+ei /usr/bin/rsync

The “belt and braces” setup Hazel recommended both locking down ssh access for the backuphelper user to ssh-keys only and locking the password on the account. To follow this advice add the following lines to /etc/ssh/sshd_config.

Match User backuphelper
 PasswordAuthentication no

And run the following command to lock the backuphelper account’s password

sudo passwd -l backuphelper

BackupPC Server Setup

The change needed here is very simple, you only need to change the user that pulls backups from your other systems. You will need to ensure that you have correctly setup the ssh keys etc for the backuphelper user.

This can be done either via the web UI or by editing the .pl config file directly. You need to change “-l root” to “-l backuphelper” for the RsyncClientCmd. An example from one of my systems is

$Conf{RsyncClientCmd} = '$sshPath -q -x -l backuphelper $host $rsyncPath $argList+';


Always be careful working with PAM, you can lock yourself out! It is worth having a root shell open “just in case” until you are familiar with the process.

A simple test is to login via ssh as backuphelper and run “rsync -avn /root” and you shouldn’t get any permission denied errors and should see a list of files.

Something to bear in mind when debugging this is that running sudo from root -> user doesn’t give the user capabilities, you need to login directly to test things.

If you want to see whether pam_cap is working when logged in you can do this:

grep CapInh /proc/$$/status

Use capsh –decode= on the resulting bit string to understand what permissions you’ve got.