Thursday, February 24, 2011

Kinect audio reverse engineering

I did some work on getting the kinect audio hardware to work as part of openkinect/libfreenect a while back. Here's some quick notes on what I've figured out and how:
Once the audio firmware has been loaded the kinect sends 524 bytes to the xbox every 1ms, every tenth packet is short (60 bytes) but potentially preceded by an empty packet. The short packets appear to be non-audio data (maybe signaling of some sort) because if you exclude them the resulting data doesn't appear to have any gaps.

The audio samples appear to be 32 bits signed each at 16khz (if you assume that sample rate then the FFT of the recorded data has the correct frequency).
The 4 channels seem to be transmitted in order left to right from the perspective of someone looking at the front of the kinect. The leftmost channel is transmitted first. 256 samples of each channel are transmitted before switching to the next channel.
if you stitch together disparate 256 sample blocks to reconstruct a given channel the data appears to be continuous. The plot below shows the captured 4-channel audio stream with the channels labeled from left to right as 1,2,3,4. You can see that the leftmost channel has the greatest amplitude corresponding to the fact that the speaker was placed closest to the leftmost microphone. I repeated the test with a speaker near the rightmost microphone and, as expected, channel 4 became the strongest.

I was able to determine the information in the last paragraph by synthesizing a 700Hz sine wave in matlab and then playing it back at the kinect with a speaker nearest the leftmost microphone (as seen from the front of the kinect). I then captured the data stream coming back from the kinect while I played the sine wave using a beagle USB sniffer. I extracted the 524 byte blocks I suspected to be audio from the beagle dumps and then post processed them with a series of shell scripts before reading them into matlab and plotting the FFT of this audio as seen below:

The frequency shown by the FFT is correctly 700Hz(approx.) This suggests that my interpretation of the audio format is correct.

Firmware loading process

I've managed to duplicate so far what I think is most of the init sequence-
I send all the same control transfers and bulk transfers as the Xbox,
as far as I can tell. My beagle480 confirms that I mirror the Xbox behavior for the most part. After completing a series of 512 byte bulk-out transfers which I
assume is some sort of bootstrapping firmware upload, the audio device
re-enumerates, I wait for that to happen and open the new audio
device, then send some more control and bulk transfers. So far ,so
good, this all follows what I see in the Xbox logs. At this point the
Xbox appears to send 12 cycles of ( 1 xfer: 0 byte iso IN, 8 xfers 4
bytes out) which I also duplicate perfectly. Now, the final step is a
very long stream of (1 xfer: 0 byte iso IN, 8 xfers 76 bytes out)
before eventually those 0 byte IN transfers become 524 bytes
transfers. Unfortunately it seems the content of those 76 byte OUT
transfers must matter because after trying all zeros I never get any
data back in my IN transfers (even after >5000 IN transfers). I have
some scripts I'll use to try generate code for all those OUT transfers
directly from the .tdc files.