Fat Secret calorie counter access with python

This post complements my earlier discussion of the Withings scale API here:
http://www.keyboardmods.com/2012/05/withings-wifi-scale-python-dashboard.html
I've written a convenience library in python that simplifies the process of retrieving protein, carbohydrate, fat, and calorie data from Fatsecret along with several other features from the REST API. To make use of the library you'll need to register as a developer at the link below:
http://platform.fatsecret.com/api/Default.aspx?screen=r

After registering you will be given a consumer/API key and secret which you will need to use my python library. Once you have your key and secret, download the python library with the following command:

git clone https://github.com/trtg/pyfatsecret.git

The library comes with the example below that demonstrates its usage. To use the example, replace the strings 'your_key_goes_here' and 'your_secret_goes_here' with the key and secret you were assigned after registering as a developer.
Note that the example above makes use of pandas to plot timeseries. You can get the latest pandas with this command:

git clone https://github.com/pydata/pandas.git

I've tested the example above with the stable 0.7.3 release as well.
Once you paste your key and secret into place and install pandas you can run the example as follows:

python example.py

Assuming you already have some data recorded with fatsecret, you should get a plot like the one below after running (click on the image to see a larger copy):

Withings wifi scale python dashboard using matplotlib and pandas

The Withings wifi scale is a great bit of hardware, much more visually appealing than my handmade equivalent and wifi makes it standalone, also a big advantage over my bluetooth scale. I've been using the scale to log my weight and using fatsecret to track caloric intake for a while now and decided it would be nice to make a dashboard integrating data from both sources. The plots on Withings' website leave much to be desired. Thankfully, both Withings and fatsecret have APIs which make third party apps possible. Below I'll show some code that uses the rauth library to access both services. rauth provides OAuth 1.0/a, 2.0, and Ofly consumer support. You can install rauth using easy_install:
sudo easy_install rauth
Before using the Withings and fatsecret APIs, you'll need to register as a developer, you can do that by following the links below:
Withings developer registration:
https://oauth.withings.com/en/partner/add

If you just want to extend the code I provide further down you can fill an arbitrary value for the "Application Website" and "Organization" fields and leave the Callback URL blank. Otherwise, fill in those fields according to your needs. Once you've registered as a developer with both services you will have your Consumer Key and Shared Secret which you will need to send API requests. Now grab my python code from github:

git clone https://github.com/trtg/pywithings.git

The git repo has two files: withings.py(the actual library itself) and example.py (seen below). Open up example.py and fill in your consumer key, consumer secret, and the email address you used to register with Withings as indicated. Using the library is pretty straightforward: instantiate a withings object and then just call get_weights() to retrieve a list of weight measurements and an associated list of dates (in seconds since the epoch format) which you can then use however you'd like. Note that the very first time you run example.py you will be prompted to authorize the app to access your withings account by pasting a URL into your browser. Once you go to that URL and login, you will see a screen like this:

After you click "allow" you will see an oauth token string as shown below. Copy just the part after "oauth_verifier=" , in this case PZBDNyyuxDnGkeMxccY.

Paste the "PZ..." string into your terminal where prompted, hit enter, and you should get a graph like the one below.(Click on the graph to see a larger version) I used the python package pandas to treat the weight data as a time series and get nice date labels on the x-axis as well as to plot the smooth red rolling average of the data.

In an upcoming post, I'll discuss similar code that retrieves data from fatsecret to generate plots like the one below:

Gymnastics tap swing trainer

Timing is one of the things I've found most difficult to pick up while learning basic gymnastics. This is especially true when it comes to high bar/strap bar giants and tap swings. If I just blindly practice, I tend to tap far too early. For back giants, the tap is the moment when you transition from a hollow to an arch then back to a hollow again. To facilitate switching from hollow to arch at the right time I built the device you see in the video below. I suspect it would also work well for kip timing (it would tell you when to bring toes to the bar). It's basically a beam break sensor, like you might have at the bottom of your garage door, except it runs on batteries and beeps whenever something crosses between the flashlight and sensor. The idea is that you place the flashlight and sensor on either side of the high-bar at the location where you should ideally tap and then as you swing you just wait to hear the beep before switching from hollow to arch.



Here you can see and hear the tap trainer in action- I know my form is terrible, the video is just to show you how the device works. That high pitched beep you hear every time I approach maybe 30 degrees away from vertical is the sensor detecting me crossing its path and then beeping.

Kinect speech recognition in linux

Audio support is now part of libfreenect. Additionally it is now possible to load the microsoft SDK version of the audio firmware from linux courtesy of a utility called kinect_upload_fw written by Antonio Ospite.This version of the firmware makes the kinect appear to your computer as a standard USB microphone.
This means you can now record audio using your kinect, but that's not all that interesting in and of itself. Linux support for speech recognition at this point is not all that great. It is possible to run dragon naturallyspeaking via wine or to use the sphinx project (after much training), but neither of those approaches really appealed to me for simple voice commands (as opposed to dictation). The google android project happens to include a speech recognizer from Nuance which by default is meant to be built for an ARM target, like your phone. After extensive hacking around the build system I was able to instead build for an x86 target, like your desktop. Now, you can combine these two things- kinect array microphone + android voice recognition to do some more interesting things, i.e. toggle hand tracking on and off via voice.



How to get started:

1) Check if you have the "unbuffer" application which is part of the linux scripting language called expect:

which unbuffer

If the above command comes up empty you should download a copy of unbuffer from the link here:
http://dl.dropbox.com/u/11217419/unbuffer

copy unbuffer to a directory that is in your path, like /usr/local/bin or ~/bin

2)Download my precompiled version of the srec subproject from here:
http://dl.dropbox.com/u/11217419/srec_kinect.tgz

3)save the tarball from step 1 in a convenient directory then unpack it with this command:
tar xfz srec_kinect.tgz

4)switch into the subdirectory where I've placed some convenience scripts:
cd srec/config/en.us

5) Open a second terminal and in that second terminal also switch into srec/config/en.us

6) In the first terminal execute
./run_SRecTestAudio.sh
and in the other terminal execute
cat speech_fifo

7) try speaking into your microphone and wait for recognition results to appear in both terminals. Note that the vocabulary as configured at this point is very small- words like up,down,left,right and the numbers from 1-9 should be recognized properly.

Integrating the kinect:
1)Acquire Antonio Ospite's firmware tools like so:
git clone http://git.ao2.it/kinect-audio-setup.git/

2)move into the kinect-audio-setup subdirectory:
cd kinect-audio-setup

3)build kinect_upload_fw as root:
make install

4)Fetch and extract the microsoft kinect SDK audio firmware (depending on your directory permissions, this may also need to be run as root):
./kinect_fetch_fw /lib/firmware/kinect

This will extract the firmware to this location by default:
/lib/firmware/kinect/UACFirmware.C9C6E852_35A3_41DC_A57D_BDDEB43DFD04

5)Upload the newly extracted firmware to the kinect:
kinect_upload_fw /lib/firmware/kinect/UACFirmware.C9C6E852_35A3_41DC_A57D_BDDEB43DFD04

6)Check for a new USB audio device in your dmesg output

7)Configure the kinect USB audio device to be your primary microphone input and
try out run_SRecTestAudio.sh again as described earlier.


Additional Notes:

I unfortunately no longer remember all the changes I had to make in order for the srec project within android build for x86. Perhaps someone with better knowledge of the android build system can chime in at the comments below. In the interim, use the precompiled copy that I have linked above, just be aware that it is old, I think it dates back to the froyo branch of android or earlier (I compiled it a long time ago). If you want to take a shot at building the latest srec yourself, check out the android source code then look under external/srec/

The run_SRecTestAudio.sh script sets up the speech recognizer to run on live audio and pipes the recognition results to a fifo in the same directory called speech_fifo. Running cat in the second terminal lets you read out the recognition results as they arrive. Instead of cat you could alternatively have whatever programs needs recognition results read from the fifo and act accordingly. Unbuffer is used to make sure you see recognition results right away rather than waiting for the speech_fifo to fill up.

The srec recognizer does not require any training but has certain limitations. The most significant limitation is the vocabulary it can recognize. The larger the vocabulary you specify, the less accurate the recognition results will likely be. As a result this recognizer is best used for a small set of frequently used voice commands. Under srec/config/en.us/grammars/ there are a number of .grxml files which define what words the recognizer can understand. You can define your own simple grammar (.grxml) here which, for example, only recognizes the digits on a phone keypad. To do this you can follow the syntax of any of the other .grxml files in the directory and then execute run_compile_grammars.sh which will produce a .g2g file from the .grxml file. There is also a voicetag/texttag file with extension .tcp which needs to point to the g2g file of your choice. You can find the .tcp files under the srec/config/en.us/tcp directory. run_SRecTestAudio.sh points to a tcp file which you can specify.

Kinect audio reverse engineering

I did some work on getting the kinect audio hardware to work as part of openkinect/libfreenect a while back. Here's some quick notes on what I've figured out and how:
Once the audio firmware has been loaded the kinect sends 524 bytes to the xbox every 1ms, every tenth packet is short (60 bytes) but potentially preceded by an empty packet. The short packets appear to be non-audio data (maybe signaling of some sort) because if you exclude them the resulting data doesn't appear to have any gaps.

The audio samples appear to be 32 bits signed each at 16khz (if you assume that sample rate then the FFT of the recorded data has the correct frequency).
The 4 channels seem to be transmitted in order left to right from the perspective of someone looking at the front of the kinect. The leftmost channel is transmitted first. 256 samples of each channel are transmitted before switching to the next channel.
if you stitch together disparate 256 sample blocks to reconstruct a given channel the data appears to be continuous. The plot below shows the captured 4-channel audio stream with the channels labeled from left to right as 1,2,3,4. You can see that the leftmost channel has the greatest amplitude corresponding to the fact that the speaker was placed closest to the leftmost microphone. I repeated the test with a speaker near the rightmost microphone and, as expected, channel 4 became the strongest.



I was able to determine the information in the last paragraph by synthesizing a 700Hz sine wave in matlab and then playing it back at the kinect with a speaker nearest the leftmost microphone (as seen from the front of the kinect). I then captured the data stream coming back from the kinect while I played the sine wave using a beagle USB sniffer. I extracted the 524 byte blocks I suspected to be audio from the beagle dumps and then post processed them with a series of shell scripts before reading them into matlab and plotting the FFT of this audio as seen below:



The frequency shown by the FFT is correctly 700Hz(approx.) This suggests that my interpretation of the audio format is correct.

Firmware loading process

I've managed to duplicate so far what I think is most of the init sequence-
I send all the same control transfers and bulk transfers as the Xbox,
as far as I can tell. My beagle480 confirms that I mirror the Xbox behavior for the most part. After completing a series of 512 byte bulk-out transfers which I
assume is some sort of bootstrapping firmware upload, the audio device
re-enumerates, I wait for that to happen and open the new audio
device, then send some more control and bulk transfers. So far ,so
good, this all follows what I see in the Xbox logs. At this point the
Xbox appears to send 12 cycles of ( 1 xfer: 0 byte iso IN, 8 xfers 4
bytes out) which I also duplicate perfectly. Now, the final step is a
very long stream of (1 xfer: 0 byte iso IN, 8 xfers 76 bytes out)
before eventually those 0 byte IN transfers become 524 bytes
transfers. Unfortunately it seems the content of those 76 byte OUT
transfers must matter because after trying all zeros I never get any
data back in my IN transfers (even after >5000 IN transfers). I have
some scripts I'll use to try generate code for all those OUT transfers
directly from the .tdc files.

HOWTO: use the kinect as a mouse in linux

In an earlier post I explained how to get PrimeSense's NITE up and running and how to use the samples they provided. Now some people might be thinking "cool, but how can I use this?" I thought using NITE hand tracking to control the cursor would be a good and simple demonstration.

The linux kernel provides a means to create userspace input drivers using a feature called uinput. If you compile your kernel with uinput enabled as a module you can then simply:
modprobe uinput
to load the uinput module. Once the module is loaded you can use the piece of code I've embedded below to convert the coordinates output by the NITE code into actual mouse/cursor movement. In short:

(1) download the code below
(2) save it as ~/kinect/NITE/Nite-1.3.0.17/Samples/SingleControl/main.cpp (you might want to back up the original)
(3)cd ~/kinect/NITE/Nite-1.3.0.17 && make
(4)Note: do the following as root or using sudo

~/kinect/NITE/Nite-1.3.0.17/Samples/Bin/Sample-SingleControl

(5)Perform a focus gesture to start the hand tracking (check out my video above to see how to do that)

At this point you should be able to do what I do in the video above. You can also extend the code to generate mouse clicks, keystrokes, etc. Have fun.
At some point dropbox ate the public link for the source code I was using before and in the process of restoring from an old backup the formatting of the code below got a bit mangled... in any case here's a gist of what I salvaged:

HOWTO: Kinect + OpenNI/NITE skeleton tracking and gesture recognition in gentoo

Thanks to the folks at PrimeSense libraries are now available for skeleton tracking and gesture recognition.
UPDATE: Check here if you've gotten NITE working and want to try using the kinect as a Minority Report style mouse.
UPDATE:I've added a description of how to track multiple hands under the Sample-PointViewer description.

Here's how I got things working in gentoo:


(1) mkdir ~/kinect && cd ~/kinect
(2) git clone https://github.com/OpenNI/OpenNI.git
(3) cd OpenNI/Platform/Linux-x86/Build
(4) make && sudo make install
(5) cd ~/kinect/
(6) git clone https://github.com/boilerbots/Sensor.git
(7) cd Sensor
(8) git checkout kinect
(9) cd Platform/Linux-x86/Build
(10) make && sudo make install
(11) go to this page at openNI to download the latest NITE release for your platform: NITE download page or for the impatient:
32-bit
64-bit
UPDATE: download links now point to openNI and should work again
(12)Save the NITE tarball to ~/kinect and untar it
(13) cd ~/kinect/NITE/Nite-1.3.0.17/Data
(14)Open Sample-User.xml and replace the existing License line with the line below:
NOTE: this is case sensitive!

< License vendor="PrimeSense" key="0KOIk2JeIBYClPWVnMoRKn5cdY4="/>

(15)Repeat step 14 for Sample-Scene.xml and Sample-Tracking.xml
(16)Open Sample-User.xml and replace the existing MapOutputMode line with the line below.
NOTE: this is case sensitive!

< MapOutputMode xRes="640" yRes="480" FPS="30"/>

(17)Repeat step 16 for Sample-Scene.xml and Sample-Tracking.xml
(18)niLicense PrimeSense 0KOIk2JeIBYClPWVnMoRKn5cdY4=
(19)cd ~/kinect/NITE/Nite-1.3.0.17/
(20)sudo ./install.bash
(21)make && sudo make install
(22)cd ~/kinect/NITE/Nite-1.3.0.17/Samples/Bin

Now finally you should be sitting in a directory with all the sample binaries that you can play with. Here's what they should look like:

Sample-TrackPad:
This app will track your hand and show it's relative position on a grid. Run it and wave your hand,one of the squares on the grid should turn yellow to indicate your hand's location as seen below:



you should also get some debug output in your console:



Sample-Players
This app demonstrates the skeletal tracking. After starting it up, move around or wave until your body changes to blue (subsequent players will be other colors, e.g. player 2 is green, 3 yellow,etc.). At this point your viewer window should look vaguely like this:



and you should see something like this in your console:
Look for pose
Found pose "Psi" for user 1


Now, hold your arms out to your sides bent 90 degrees at the elbows as shown below until a skeleton is overlayed on the image of your body:


At this point something like this should have appeared in your console:


Calibration started
Calibration done [1] successfully
Writing 217.596 50 50 78.4388 64.6762
Matching for existing calibration
Read 217.596 50 50 78.4388 64.6762

Sample-SingleControl
This seems to do some sort of gesture recognition and dynamically adjusts the camera resolution, so it's probably zooming in on an area of interest. When it starts out it asks you to perform a focus gesture. The NITE documentation doesn't seem to define what this would be but simply sticking one hand out in front of you seems to make it happy and you'll see the following output:



SamplePointViewer
This app does handtracking. UPDATE: to allow multiple hands to be tracked you will need to edit /usr/etc/primesense/XnVHandGenerator/Nite.ini by uncommenting the two config parameters it contains. Basically remove the semicolons at the start of each line so that Nite.ini looks like this:

[HandTrackerManager]
AllowMultipleHands=1
TrackAdditionalHands=1


To persistently track different hands in your code you can make use of the XnVHandPointContext.nID in your OnPointUpdate callback.



Sample-Boxes
This example allows you to click one of three boxes, your hand motion is tracked by a slider and depending on the context, up, left, right gestures will be recognized.



Sample-CircleControl
Wave to make the border of the window turn green. Then I think you need to send a focus gesture and then if you trace out a circle in the air with your hand the onscreen circle will follow your hand as seen below. In other words if you draw a clockwise circle in the air, the clock hand will also spin clockwise and vice versa. For some reason, this appears to be annoyingly inconsistent.



Sample-SceneAnalysis
This seems to just do player detection without skeleton tracking: