Lytro Unlock – Making a bad camera slightly better

TL;DR; summary

I’ve recently spent some time playing with and reverse engineering this curious piece of tech that was a first consumer oriented,though odd looking, lightfield camera called Lytro. Killer feature of this new technology was the ability to refocus the image after it was taken!
The bad side was that the software was pretty bad, the camera was trying to solve a problem that didn’t exist and the whole endeavor mostly failed.

This project documents and implements a python library that unlocks and makes available a few interesting features that weren’t ever made available through official software. Among those are:

• full remote camera parameters control
• Live View streaming
• Debug console
• Custom code execution

Although it failed as a commercial product, lytro camera has some pretty nifty tech (only one of which is very high optical zoom). It is my hope that somebody out there has a cool idea that would benefit from having full software control over the camera.
If you are interested in the details, read on. Otherwise, feel free to play with supporting software under lytroctrl.

No physical modifications to the cameras are neccessary to play with this. The unlocking and subsequent commands work over WiFi.
However, there’s a chance it can brick your camera (it’s kind of a brick already) and it will most definitely void your waranty! Be warned!

The Why – Start

Anybody remember these weird rectangles from the photo above? They were all over tech news websites almost a decade ago. They were supposed to usher in the new age of photography. Some of you might have them tucked away in some drawer as a reminder of wasted money. The product promised so much, yet what it delivered in reality was oh so disappointing.

Why did I get interested in this? Few months ago, an interesting twitter thread caught my eye. It was written by Warren Craddock and was talking about tech projects can have so much momentum that people can ignore the obvious flaws and showstoppers for a long time before they are abandoned (note: tweets have been deleted, but you can dig them up in the archive – they have some interesting insights). One of the projects he talked about was a Lytro camera. Flaws being that , due to laws of physics and optics, most of the benefits of the technology (refocus, paralax) would be of no practical use to general public. Long story short, after developing a second generation of the LightField camera called Illum (this time in a more familiar DSLR-like package) company soon went bust.

Naturally, the first question that popped into my mind was: “I wonder how cheap those are on eBay right now?”
Sure enough, if you are patient, you can easily pick them up for anywhere from $20 to $40 dollars. A great discount from the original retail price.
I decided to procure one, if for no other reason then simply to have it as an interesting relic. But the curiosity got the better of me and I soon found myself digging into the firmware.

Goals

Lets review the current state of affairs with regards to Lytro and set out some goals.
While the company is gone, most of the online materials, firmware and software has been kindly archived in different places online.
The camera itself has very little in terms of controls, yet you are supposed to use it as a standalone device. It’s not permanently tethered to a phone (although there is/was an app for it at some point). One would expect that, when connected to a computer, the camera would show up as a regular mass storage device, but that is not the case. Rather, you have to use Lytro software to download and process the images. Windows and OS X versions are available, but look like they were already outdated 10 years ago and leave a lot to be desired.
Newer firmware versions support WiFi access point mode, but the official software doesn’t make much use of it. Maybe the app did once, but it doesn’t matter. No official API or documentation was ever provided.

There were a couple of community and 3rd party projects that aimed to replicate the official software and did a very good job on documenting how things work. Highlights of those are :

• lytro meldown by Jan Kučera which was of immense help as it documents publicly known parts of the communication protocol, has an archive of previous firmware versions and still runs a firmware update service.
• Lyli – linux based implementation of USB communication protocol

While both of these provide good starting points, they don’t offer much benefit over the official software, only a few extra features.

Just looking at a product, without considering existing software, I thought it would be a lot more useful if it had features that a normal webcam would have. With that in mind, I set out to see if implementing the following was possible:

• control zoom level through software
• Control focus point through software
• Take a picture on demand
  Initially, as stretch goals I was considering:
• firmware modification
• Arbitrary code execution
• Live View / video streaming

Ok, with that, lets have a specific use case we want to make possible: Aim the camera through the window, zoom to certain level and feed the live view images to computer-vision code. Whenever a squirrel is detected in view , take a photo. Then refocus on demand later. Wouldn’t that be fun?

Take it apart

First things first, lets take a look at how this thing is made.

Due to its unusual shape , the device has an unusual number of boards that are interconnected. This makes it a bit of a pain to disassemble, solder wires to test points and then reassemble but just by looking at the following photos we can identify interesting things.

Topmost board contains the sensor with the microlens array and connects further to lens controls. Lens has tiny motors and actuators that control zoom, focus, shutter and built in neutral density filter.

Next is a power/battery control board which is also where usb connection is located.

Beneath the battery is the main SoC board which is based around a MIPS MCU called Coach (Camera On A Chip) from Zoran corporation. This SoC seems to be very common with dashcam manufacturers from the same era. On the flipside, we can see Samsung’s flash and other ICs. Certain labeled test points would suggest a JTAG interface to be present.

Finally, the last board connects to the display as well as the capacitive touch sensor that controls zoom. It is on this board that we find a peculiar looking unpopulated connector pad. Basic testing quickly reveals UART pins like shown, but all we get is a short boot message and no input or echo…

Lets make a nice breakout of those pins for later use just in case.

The firmware

So we have a serial port, but it doesn’t give us anything useful (yet!). Let’s check out the firmware to see if we should even expect anything on the serial output.
Luckily for us, “Lytro Meltdown” has archived all the public firmware versions at : http://optics.miloush.net/lytro/TheResources.aspx

Firmware update seems to be complete and not incremental, which is a good thing. It starts with a simple JSON text that describes the actual data that follows. It appears that the firmware is just flat memory contents whithout any disernable file system. Binwalk reveals the following:

Remember, ELF files don’t imply that we are dealing with Linux. And indeed, in this case, ELF format is used for memory mapping and binary loading purposes. All the binaries seem to be mapped to their respective addresses and cross-reference each other.

2940.so:   ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1 (SYSV), statically linked, no section header
2A0540.so: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1 (SYSV), statically linked, no section header
3D4594.so: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1 (SYSV), statically linked, no section header
5F6C68.so: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1 (SYSV), statically linked, no section header
63DB40.so: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), statically linked, not stripped
641B40.so: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), statically linked, not stripped
645B40.so: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), statically linked, not stripped
649B40.so: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), statically linked, not stripped
940.so:    ELF 32-bit LSB executable, MIPS, MIPS-II version 1 (SYSV), statically linked, stripped

class CommandClass(Enum):
	CMD_CTRL = 0xC0
	CMD_FW = 0xC1
	CMD_LOAD = 0xC2
	CMD_CLR = 0xC3
	CMD_READ = 0xC4
	CMD_FW_UP = 0xC5
	CMD_QUERY = 0xC6


class CtrlCmd(Enum):
	CMD_CLASS = CommandClass.CMD_CTRL
	TAKE_PHOTO = 0x00
	DELETE = 0x01
	UPDATE_FW = 0x02
	REBOOT = 0x03
	SET_TIME = 0x04
	UNK1 = 0x05 # something with files
	UNK2 = 0x06 # delete all ? 
	SET_SHARE_MODE_SSID = 0x07
	SET_SYNC_MODE_SSID = 0x08
	SET_SHARE_MODE_PASS = 0x09
	SET_SYNC_MODE_PASS = 0x0A
	SET_SHARE_MODE_AUTH = 0x0B
	MANUAL_CTRL = 0x0C
	WIFI_OFF = 0x0D
	INJECT_UART = 0xFE
	EXEC_CMD = 0xFF

class FwCmd(Enum):
	CMD_CLASS = CommandClass.CMD_FW
	SET_FW_SIZE = 0x00

class LoadCmd(Enum):
	CMD_CLASS = CommandClass.CMD_LOAD
	CAMERA_INFO = 0x00
	FILE = 0x01
	PHOTO_LIST = 0x02
	UNK1 = 0x03
	UNUSED = 0x04
	PHOTO = 0x05
	CALLIBRATION = 0x06
	RAW  = 0x0a
	CRASHES = 0x0b

class ClearCmd(Enum):
	CMD_CLASS = CommandClass.CMD_CLR
	CLEAR_BUFFERS = 0x00

class ReadCmd(Enum):
	CMD_CLASS = CommandClass.CMD_READ
	READ = 0x00

class FirmwareUploadCmd(Enum):
	CMD_CLASS = CommandClass.CMD_FW_UP
	UPLOAD_CHUNK = 0x00

class QueryCmd(Enum):
	CMD_CLASS = CommandClass.CMD_QUERY
	CONTENT_LENGTH = 0x00
	UNK1 = 0x01
	READ_UART = 0x02
	CAMERA_TIME = 0x03
	RECOVER_CALLIBRATION  = 0x04
	READ_GLOB_80020a60 = 0x05
	BATTERY = 0x06
	SETTINGS = 0x0A

	usage: shell.py [-h] [-v] [-a ADDRESS] [-p PORT] [-l] [-s]

Lytro control shell

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         output includes debug logs
  -a ADDRESS, --address ADDRESS
                        ip address
  -p PORT, --port PORT  destination port
  -l, --no-unlock       don't perform unlocking
  -s, --no-keepalive    don't start keepalive connection (camera will sleep due to inactivity)

	con = connections.TcpConnection(args.address,int(args.port))
con.connect()
if not args.no_unlock:
	print("Unlocking...")
	tr = transactions.GetCameraInfoTransact(con)
	ci = tr.transact()
	camera_serial = ci.serial_num
	transactions.UnlockTransact(con,camera_serial).unlock()
	print("Unlocked!")

print("Do the lens dance")
transactions.ExecTransact(con,"rfitamron wiggle").transact()
time.sleep(3)
print("Zooming to x3.14...")
transactions.SetZoom(con).transact(payload=b"3.14")
time.sleep(1)
print("Taking photo")
transactions.ExecTransact(con,"rficapture").transact()

transactions.UDPLiveView(con).transact(enable=True,cratio=2,fps=1)
utils.keep_alive(con)