I’ve recently spent some time playing with and reverse engineering this curious piece of tech that was a first consumer oriented,though odd looking, lightfield camera called Lytro. Killer feature of this new technology was the ability to refocus the image after it was taken!
The bad side was that the software was pretty bad, the camera was trying to solve a problem that didn’t exist and the whole endeavor mostly failed.
This project documents and implements a python library that unlocks and makes available a few interesting features that weren’t ever made available through official software. Among those are:
- full remote camera parameters control
- Live View streaming
- Debug console
- Custom code execution
Although it failed as a commercial product, lytro camera has some pretty nifty tech (only one of which is very high optical zoom). It is my hope that somebody out there has a cool idea that would benefit from having full software control over the camera.
If you are interested in the details, read on. Otherwise, feel free to play with supporting software under lytroctrl.
No physical modifications to the cameras are neccessary to play with this. The unlocking and subsequent commands work over WiFi.
However, there’s a chance it can brick your camera (it’s kind of a brick already) and it will most definitely void your waranty! Be warned!
The Why – Start
Anybody remember these weird rectangles from the photo above? They were all over tech news websites almost a decade ago. They were supposed to usher in the new age of photography. Some of you might have them tucked away in some drawer as a reminder of wasted money. The product promised so much, yet what it delivered in reality was oh so disappointing.
Why did I get interested in this? Few months ago, an interesting twitter thread caught my eye. It was written by Warren Craddock and was talking about tech projects can have so much momentum that people can ignore the obvious flaws and showstoppers for a long time before they are abandoned (note: tweets have been deleted, but you can dig them up in the archive – they have some interesting insights). One of the projects he talked about was a Lytro camera. Flaws being that , due to laws of physics and optics, most of the benefits of the technology (refocus, paralax) would be of no practical use to general public. Long story short, after developing a second generation of the LightField camera called Illum (this time in a more familiar DSLR-like package) company soon went bust.
Naturally, the first question that popped into my mind was: “I wonder how cheap those are on eBay right now?”
Sure enough, if you are patient, you can easily pick them up for anywhere from $20 to $40 dollars. A great discount from the original retail price.
I decided to procure one, if for no other reason then simply to have it as an interesting relic. But the curiosity got the better of me and I soon found myself digging into the firmware.
Lets review the current state of affairs with regards to Lytro and set out some goals.
While the company is gone, most of the online materials, firmware and software has been kindly archived in different places online.
The camera itself has very little in terms of controls, yet you are supposed to use it as a standalone device. It’s not permanently tethered to a phone (although there is/was an app for it at some point). One would expect that, when connected to a computer, the camera would show up as a regular mass storage device, but that is not the case. Rather, you have to use Lytro software to download and process the images. Windows and OS X versions are available, but look like they were already outdated 10 years ago and leave a lot to be desired.
Newer firmware versions support WiFi access point mode, but the official software doesn’t make much use of it. Maybe the app did once, but it doesn’t matter. No official API or documentation was ever provided.
There were a couple of community and 3rd party projects that aimed to replicate the official software and did a very good job on documenting how things work. Highlights of those are :
- lytro meldown by Jan Kučera which was of immense help as it documents publicly known parts of the communication protocol, has an archive of previous firmware versions and still runs a firmware update service.
- Lyli – linux based implementation of USB communication protocol
While both of these provide good starting points, they don’t offer much benefit over the official software, only a few extra features.
Just looking at a product, without considering existing software, I thought it would be a lot more useful if it had features that a normal webcam would have. With that in mind, I set out to see if implementing the following was possible:
- control zoom level through software
- Control focus point through software
- Take a picture on demand
Initially, as stretch goals I was considering:
- firmware modification
- Arbitrary code execution
- Live View / video streaming
Ok, with that, lets have a specific use case we want to make possible: Aim the camera through the window, zoom to certain level and feed the live view images to computer-vision code. Whenever a squirrel is detected in view , take a photo. Then refocus on demand later. Wouldn’t that be fun?
Take it apart
First things first, lets take a look at how this thing is made.
Due to its unusual shape , the device has an unusual number of boards that are interconnected. This makes it a bit of a pain to disassemble, solder wires to test points and then reassemble but just by looking at the following photos we can identify interesting things.
Topmost board contains the sensor with the microlens array and connects further to lens controls. Lens has tiny motors and actuators that control zoom, focus, shutter and built in neutral density filter.
Next is a power/battery control board which is also where usb connection is located.
Beneath the battery is the main SoC board which is based around a MIPS MCU called Coach (Camera On A Chip) from Zoran corporation. This SoC seems to be very common with dashcam manufacturers from the same era. On the flipside, we can see Samsung’s flash and other ICs. Certain labeled test points would suggest a JTAG interface to be present.
Finally, the last board connects to the display as well as the capacitive touch sensor that controls zoom. It is on this board that we find a peculiar looking unpopulated connector pad. Basic testing quickly reveals UART pins like shown, but all we get is a short boot message and no input or echo…
Lets make a nice breakout of those pins for later use just in case.
So we have a serial port, but it doesn’t give us anything useful (yet!). Let’s check out the firmware to see if we should even expect anything on the serial output.
Luckily for us, “Lytro Meltdown” has archived all the public firmware versions at : http://optics.miloush.net/lytro/TheResources.aspx
Firmware update seems to be complete and not incremental, which is a good thing. It starts with a simple JSON text that describes the actual data that follows. It appears that the firmware is just flat memory contents whithout any disernable file system. Binwalk reveals the following:
Remember, ELF files don’t imply that we are dealing with Linux. And indeed, in this case, ELF format is used for memory mapping and binary loading purposes. All the binaries seem to be mapped to their respective addresses and cross-reference each other.
Even though we don’t know the exact size of the binaries, we can assume they are contiguous in memory and just extract the maximum possible size. That gives us the following ELF files:
2940.so: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1 (SYSV), statically linked, no section header 2A0540.so: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1 (SYSV), statically linked, no section header 3D4594.so: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1 (SYSV), statically linked, no section header 5F6C68.so: ELF 32-bit LSB executable, MIPS, MIPS32 rel2 version 1 (SYSV), statically linked, no section header 63DB40.so: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), statically linked, not stripped 641B40.so: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), statically linked, not stripped 645B40.so: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), statically linked, not stripped 649B40.so: ELF 32-bit LSB executable, MIPS, MIPS32 version 1 (SYSV), statically linked, not stripped 940.so: ELF 32-bit LSB executable, MIPS, MIPS-II version 1 (SYSV), statically linked, stripped
Examining the first one reveals what appears to be a bootloader, or at least a stage of it. We can see the same strings we observed in UART output.
Binaries that follow are bigger and appear to contain a lot more functionality.
Randomly stumbling through the strings in
2A0540 reveals the following:
This definitely appears to be some sort of command palette with command names, descriptions and associated target functions. This is great news! Already I can see some that look very interesting. The question is, how do we get to them? Remeber, no response over UART.
With UART unresponsive, but encouraged by what we saw in the firmware, we turn to other was of interacting with the camera. That is, USB and WiFi.
Again, thanks to docs, we have a pretty good start when it comes to wifi communication. Similarly, from Lyli project, we can see that WiFi communication is actually based on USB protocol and mostly follows the same protocol. Only that USB is actually based around SCSI commands, while WiFi is direct over TCP/IP.
Lytro meltdown doccuments the following set of commands:
- load hardware info (
C2 00 00)
- load file (
C2 00 01)
- load picture list (
C2 00 02)
- load picture (
C2 00 05)
- load calibration data (
C2 00 06)
- load compressed raw picture (
C2 00 07)
- download command (
- query command (
- query camera time (
C6 00 03)
- query battery level (
C6 00 06)
- take a picture (
C0 00 00)
- set camera time (
C0 00 04)
Some of these look incomplete and I am willing to bet there are more so our next step is to go back to the firmware to try to find where these are being processed.
Command interpreter – Finding our way around
Now we are faced with a conundrum. We have this giant piece of code, all of these binaries from the firmware, and we want to find the small chunk of code that’s parsing incoming Wifi or USB data in order to locate the code that corresponds to commands observed from official software.
This is a common problem in reverse engineering that I like to call localization. It’s much harder to figure out what the code does without knowing the context it’s being executed in. This can often be simply solved with a debugger, but in this case we don’t have such luxury.
Unusual or rare constants are a reverser’s friend. Those can be unique cryptographic IVs, but also something much less exotic. Consider that there are 3600 seconds in an hour. That seems like a round number in decimal, but is a somewhat unusual looking constant in hex. As such, there’s a chance it will be somewhat rare in the binary you are looking at.
On the other hand, SetTime command from above expects camera time in a very specific format. We can guess that at some point during its execution, it would need to convert seconds to hours or vice versa. Searching for seconds-in-hour (3600 or 0xe10)constant in the binary reveals only a handful of locations which we can manually investigate. And sure enough, one of them definitely look like time/date formatting code.
Backtracking through references quickly leads us to code that looks awfully like what we’d expect a command interpreter to look like. A series of switch statements with command handlers.
Success! Now we have our bearing in the firmware and can start exploring and make guesses about functionality with more confidence.
Command classes, commands and payloads
All in all, by studying the command interpreter code we can deduce the following (mostly) complete set of commands. All of them match with what was previously documented and observed, and there’s quite a few hidden an unknown ones:
class CommandClass(Enum): CMD_CTRL = 0xC0 CMD_FW = 0xC1 CMD_LOAD = 0xC2 CMD_CLR = 0xC3 CMD_READ = 0xC4 CMD_FW_UP = 0xC5 CMD_QUERY = 0xC6 class CtrlCmd(Enum): CMD_CLASS = CommandClass.CMD_CTRL TAKE_PHOTO = 0x00 DELETE = 0x01 UPDATE_FW = 0x02 REBOOT = 0x03 SET_TIME = 0x04 UNK1 = 0x05 # something with files UNK2 = 0x06 # delete all ? SET_SHARE_MODE_SSID = 0x07 SET_SYNC_MODE_SSID = 0x08 SET_SHARE_MODE_PASS = 0x09 SET_SYNC_MODE_PASS = 0x0A SET_SHARE_MODE_AUTH = 0x0B MANUAL_CTRL = 0x0C WIFI_OFF = 0x0D INJECT_UART = 0xFE EXEC_CMD = 0xFF class FwCmd(Enum): CMD_CLASS = CommandClass.CMD_FW SET_FW_SIZE = 0x00 class LoadCmd(Enum): CMD_CLASS = CommandClass.CMD_LOAD CAMERA_INFO = 0x00 FILE = 0x01 PHOTO_LIST = 0x02 UNK1 = 0x03 UNUSED = 0x04 PHOTO = 0x05 CALLIBRATION = 0x06 RAW = 0x0a CRASHES = 0x0b class ClearCmd(Enum): CMD_CLASS = CommandClass.CMD_CLR CLEAR_BUFFERS = 0x00 class ReadCmd(Enum): CMD_CLASS = CommandClass.CMD_READ READ = 0x00 class FirmwareUploadCmd(Enum): CMD_CLASS = CommandClass.CMD_FW_UP UPLOAD_CHUNK = 0x00 class QueryCmd(Enum): CMD_CLASS = CommandClass.CMD_QUERY CONTENT_LENGTH = 0x00 UNK1 = 0x01 READ_UART = 0x02 CAMERA_TIME = 0x03 RECOVER_CALLIBRATION = 0x04 READ_GLOB_80020a60 = 0x05 BATTERY = 0x06 SETTINGS = 0x0A
That’s quite a lot of extra functionality reachable over USB or WIFI! Most are self explanatory, but some I am yet to explore. However, we aren’t done yet. Remeber that UART is still locked.
Secret Unlock command
You’ll notice in the above that I’ve listed an EXEC command. It was an obvious choice to focus on and try to figure out. At first, it looked like it just expects a string to run as command, but it simply wouldn’t work. Digging deeper into it revealed something interesting. It first caught my attention because it’s clearly doing something with SHA1. That has to be something interesting!After cleaning up the decompiled code, the functionality becomes obvious.
The function takes some string from the memory, concatenates “ please” to it and calculates a SHA1 hash of it. Then it compares that hash to whatever was received as payload.
After a bit more digging, it becomes obvious that the string in memory is actually the serial number of the camera. We can easily get the serial number, so let’s calculate the expected hash, send it and see what happens:
Well, looks like that takes care of the locked serial console. In addition to unlocking the serial output, this enables command execution and a lot of other functionality. It turns out that a good number of above commands were checking this global variable before being run. Sending the correct hash sets the variable and enables the locked commands. Thanks Lytro for not making it too complex!
Commands over Wifi
Full list of commands is quoted above. Not all of them have been fully implemented in the accompanying software yet. If you find it useful, feel free to make a pull request.
To test it out, enable WiFi on the lytro camera and connect to it. Then run one or more samples from lytroctrl examples.
Some of the commands are more complex than others and some even have subcommands of their own.
Looks like two modes are supported, the AP mode that is the default, and “sync” mode where the camera connects to the specified access point. The second mode would probably be a lot more convenient to use, but I haven’t yet found a way to persist these particular settings across reboots.
Parameters are enable/disable, compression ratio and FPS specification. When Live View is enabled, a new thread is started on the camera that simply sends JPEG frames over multicast UDP . These can easily be captured by ffmpeg for example:
ffmpeg -i 'udp://220.127.116.11:10101?&localaddr=10.100.1.100' -vcodec copy output.mp4. It’s a bit unstable, and care should be taken when choosing the parameters. Check out the
LiveView.pyscript in lytroctrl
Live View above is actually implemented as a subcommand to manual control command. Subcommands are as follows :
- enable manual controls
- set creative mode
- control nd filter
- set zoom level
- set exposure
- set sensor gain
- enable live view
Command Exec and Serial Sync
As shown previously, the firmware has a sort of command shell which can be accessed thought serial console. It turns out that we can send commands to it directly through CmdExec command. Similarly, we can read the serial output buffer via SerialSync command. This enables implementing a simple shell over wifi. Check out the
shell.py script in lytroctrl to see how it works.
With ability to unlock the camera over wifi, and with combined ExecCmd and SerialSync commands there’s no need to open the camera or do any physical modifications in order to explore the built in shell. The following is the complete list of commands with their associated descriptions:
Gives full control over the lens. This one actually has a few “undocumented” subcommands that enable precise focus control, autofocus activation , zoom and everything that has anything to do with the lens (I guess the name reveals who was the manufacturer of the lens, Tamron is a well known brand).
All the camera settings can be read by issuing
rfisettingscommand which will just dump them to stdout. Similarly
rfisettingcommand can be used to change individual ones. Some don’t seem to persist over reboots, though.
Simply commands the camera to take the photo.
Tools and utils
In lytroctrl, I’ve implemented a number of commands and set a scafolding for all of them. I’ve made a couple of utilities that show how you can interact with the camera over WiFi.
usage: shell.py [-h] [-v] [-a ADDRESS] [-p PORT] [-l] [-s] Lytro control shell optional arguments: -h, --help show this help message and exit -v, --verbose output includes debug logs -a ADDRESS, --address ADDRESS ip address -p PORT, --port PORT destination port -l, --no-unlock don't perform unlocking -s, --no-keepalive don't start keepalive connection (camera will sleep due to inactivity)
A simple shell will connect to the camera over WiFi, unlock it, present you with stdout and expect commands to be sent. Whenever a command is sent, output is synced back. Note that by default, error/logging is also emited which can be annoying so it’s disabled by default, use
-v to enable that.
There’s a short script that demonstrates different functionalities by wiggling the lens, controling zoom and taking a photo:
con = connections.TcpConnection(args.address,int(args.port)) con.connect() if not args.no_unlock: print("Unlocking...") tr = transactions.GetCameraInfoTransact(con) ci = tr.transact() camera_serial = ci.serial_num transactions.UnlockTransact(con,camera_serial).unlock() print("Unlocked!") print("Do the lens dance") transactions.ExecTransact(con,"rfitamron wiggle").transact() time.sleep(3) print("Zooming to x3.14...") transactions.SetZoom(con).transact(payload=b"3.14") time.sleep(1) print("Taking photo") transactions.ExecTransact(con,"rficapture").transact()
And finaly, an example that shows how to enable live view streaming. Streaming is performed over UDP multicast, and consists simply of transmited jpegs. You can either capture those directly, or instruct ffmpeg to save them.
Keep alive function above will periodically issue a query command just to stave off the watchdog which turns of wifi to conserve battery.
Lua code execution
Firmware appears to have support for Lua scripts. This is still a TODO. It would be nice to have a documented way of executing Lua code.
And with that, we have all the components for what we set out to do. We can control zoom and focus, we can telecommand the camera to take a photo and we can stream live view which can be fed into computer vision algorithms.
There’s plenty of squirells in my backyard, but I can take much better photos of them with my regular camera, and most of them are in focus!
That being said, I had fun doing this and would love to hear if somebody has any interesting ideas that could benefit from this project. Lightfield cameras are definitely an interesting technology and this might be an affordable way to experiment with them.
Feel free to open issues to ask questions and make pull requests!