Why Does Dropbox Add a Unique ID to Every Photo?

Following on from my last post about Dropbox changing my photos, I noticed a new exif field of “Image Unique ID” embedded by Dropbox in the image.

This ID would allow Dropbox to track unique files across their storage estate to avoid duplication. Equally it could used to track the original file and who uploaded it from a cropped version posted online, especially if law enforcement turned up with legal papers and demanded access.

Think about leaked documents or protest photos, yes it’s good practise to strip the meta data out but not everyone does.

This again comes back to what Dropbox and it’s camera upload feature is doing and is it documented anywhere?

Note Google Photos does not embedded any tracking data in the exif of the image I tested by uploading and downloading it.

The Hash

The hash for IMG_7082 is 8af323e74def610b0000000000000000 which looks like a 128bit hash but with only the first 64 bits populated. I’ve tried a number of hash tools on various parts of the original but they don’t match the unique ID. I have tested just the pure image data from original and Dropbox modified images.

Reverse engineering the hash function isn’t the real issue here,  the real question is why has this ID been added?

Dropbox iPhone Camera Upload Changes Photos

When Google announced their new Photos tools I decided to give it a go and see what Google’s machine learning could extract from my 83,292 photos stretching back 15 years. I’m sure you know that Google are offering “unlimited” and “free” storage for photos so long as you allow them to optimize your photos. I’m happy with the trade-off in quality as I already manage an archive of full resolution (or so I thought) photos via f-spot and have backup arrangements for it.

Dropbox Camera Upload

I have used the Dropbox Camera Upload feature for about 18 months to get photos off my iPhone and on to my various other devices and offsite backup server. Dropbox state that “When you open the app, photos and videos from your iPhone or iPad are saved to Dropbox at their original size and quality in a private Camera Uploads folder.”

This statement hides the fact, that the Dropbox app re-compresses your photos before it uploads them. I found this out when I used the desktop backup client to seed Google Photos from my Dropbox camera folder, before activating the apps on my iPhone and iPad.

Google checksum all photos before uploading to avoid duplication. When I enabled the Google Photos app on my iOS devices to upload directly from the iOS camera roll, the app started to upload all my photos again. This led to duplicated photos and a few gigs of wasted upload bandwidth. I wanted to understand why this happened and adapt my photo work flow to avoid it happening again.

Image checksums

First of all I extracted a single photo IMG_7082 taken that day directly from my iPhone over USB. I copied the file from the DCIM folder on the phone, gaving me a 2.8MB file as my “master copy”. Checking my Dropbox “Camera Uploads” folder I found the same photo as expected had been renamed by Dropbox but unexpectedly had a different checksum and was over 1 megabyte smaller, the plot thickens!

The obvious next question was what is changing the file, so I extracted the same image file via email (sent as full resolution), iCloud and Photos on my MacBook each time it was the same size with a matching sha1 checksum. Uploading the master file to the free tier of Google Photos and then extracting it via Google Drive or the web UI did change the file but Google are upfront about that.

The Proof

I have created a github repo with all the photos I used in testing if you want to have a look at them yourself it’s here: https://github.com/TimJDFletcher/IMG_7082

Quality Change

I had a quick go at reproducing the same change in size of the image using GIMP and changing the JPEG compression level. I found that at 85% the file size was very close to the file size the both Google Photos and Dropbox produced. This is pretty crude test and is not to say this is the only compression that Google and Dropbox do.

Lessons Learned

The main lesson for me is that I should confirm how applications I rely on to move data work as advertised. I do understand why Dropbox re-compress photos as it gives a large saving in storage and bandwidth, I wish they were as upfront about this as Google are.

Google says they will optimize your photos, if you don’t like this then you can pay money to store the originals. Dropbox on the other hand say “Don’t worry about losing those once-in-a-lifetime shots, no matter what happens to your iPhone.”

Fixing the duplicates

Fixing the duplicates was fairly simple in the end, I just got a list of files uploaded from my iPhone and then deleted them from Google Drive using google-drive-ocamlfuse and a bit of shell script.

Smoothwall in a heterogeneous network

Smoothwall is a Linux based UTM appliance, combining a firewall, web proxy and content filter. I have recently implemented Smoothwall for a customer, this implementation included Single Sign On (SSO) support for both Mac OS X and Windows. I didn’t find any good documentation on SSO with Smoothwall for both Mac OS X and Windows so I’ve written up my notes.

Windows has for a long time had SSO support via NTLM, meaning that Windows can (fairly) securely and transparently log in to other systems that are joined to the same Active Directory controller. This is done with a ticket based challenge/response authentication process built into Active Directory.

Mac OS X has had support for Kerberos SSO via Kerberos tickets to various systems since 10.3, it has been through a number of revisions and changes over the years. However it’s not until Mac OS X 10.6.8 that support for Kerberos authentication to web-proxies like the guardian filter in Smoothwall was introduced.

This is the final piece in the puzzle for this customer and now both Apple Macs and Windows desktops “just work” automatically authenticating with Kerberos tickets.

Kerberos SSO requires slightly more careful configuration than NTLM. The main thing to make sure about is that you are accessing the proxy via it’s fully quallified name, ie proxy.example.com not just proxy or it’s IP address.

In this case the customer uses a proxy.pac file, which also needs to contain the proxy server’s full name. Smoothwall includes an option to enable this but it didn’t seem to work in this case so I just made my own simple .pac file and uploaded it.

The configuration on the clients was simple just set the network proxy settings to URL auto-configuration and point it at http://proxy.example.com/proxy.pac

I think Apple are telling the truth about DROPOUTJEEP

When I first listened to Jacob “@ioerror” Applebaum talk from the 30c3 conference in Berlin I was impressed with the number and variety of different tools the NSA was using to monitor and spy on everything. One tool I was especially interested in being an iOS user was DROPOUTJEEP, sold as allowing full access with a 100% success rate in attacking iPhones but only with physical access.
dropoutjeep catalog pageThe physical access part and Apple’s flat denial got me thinking and combined with some practical knowledge about how iOS security works I am fairly sure I understand what DROPOUTJEEP is. I have broken into iOS devices to recover data and pictures for people, it’s simple for older iPhones if you have physical access.
The earlier iPhones (before the 4S) have bootloader flaws allowing an attack on the phone before the iOS kernel boots and the kernel security kicks in. This type of attack allows the injection of a custom ramdisk image containing unsigned code. I used a ramdisk and injection tool that was assembled from an open source iOS exploit and includes an ssh server, and tools to brute force crack an iPhone PIN code using the iPhone’s own crypto hardware to accelerate the process.
The DROPOUTJEEP catalogue page is from 2008, when the iPhone 3G was the new thing, check out the timeline on Wikipedia. Most people who had an iPhone at this time remember the Limera1n jailbreak, which at it’s core is a boot loader attack. This same boot loader flaw is still unpatched even after nearly 4 years. The Limera1n attack works on the iPhone 3G onwards and there are earlier boot loader attacks too. The SHAtter boot loader attack, while never used in a jailbreak was much discussed.These types of bootloader attacks allowing the upload of custom root disks, give full hardware, disk and keybag access ie total ownage. Once your ramdisk is loaded you can just mount the internal storage and extract the data or use additional exploits to install custom malware. Full disk encryption is only a recent development, ie iPhone 3GS onwards ref: http://support.apple.com/kb/HT4175DROPOUTJEEP’s need for physical access tallies to my mind with a boot loader attack. Boot loader attacks are simple, reliable and quick so perfect for an NSA black bag job.

All this is not to say that Apple can’t push remote code to an iOS device, remember Apple has the signing keys and can sign any code they like. There are reports from the early days of iMessage that Apple pushed custom remote code to a stolen iPhone to disable iMessage.

So I think that Apple are telling the truth about DROPOUTJEEP, but that is not to say they don’t cooperate in other ways when warrants or national security letters are involved.