Leap Motion in Gentoo Linux

Great news for those of you with Leap Motion developer kits, today's 0.7.6 release of the SDK adds linux support. Their package targets ubuntu specifically but I was able to get it working on a gentoo system as well. Here's a preliminary write up of what I did:

1. Download the 0.7.6 SDK (if you're a Leap developer you know how to get it)
2. move the developer kit tarball to a convenient temp directory
3. tar xvzf Leap_Developer_Kit_0.7.6_3012_Linux.tar.gz
this will extract the debian packages- for gentoo we can convert to tarball in the next step using alien.
4. alien -t Leap-0.7.6-x64.deb
Note: if you're on a 32-bit system use Leap-0.7.6-x86.deb
5. tar xvzf leap-0.7.6.tgz
6. the previous step extracted usr and etc directories so now we copy the files from these directories into place:
   sudo cp -ir usr/local/* /usr/local/
   Note that I use -i flag to avoid accidentally overwriting existing files. I wound up just copying all the files into place rather than just the leap-specific ones because the Leap installer has bundled versions of QT libraries and trying to use my system QT libraries was causing a segfault.
7. sudo cp -ir etc/udev/rules.d/* /etc/udev/rules.d/
   Copying those udev rules should in theory allow you to run the Leap application without superuser privileges, but that did not work for me- I will try to figure this out later.
I'm not sure if these next 3 steps are necessary on gentoo, but just in case- they create the plugdev group if it does not exist, add you to that group, and then reload your session so your group membership is updated without having to logout/login.
8. groupadd plugdev
9. sudo usermod -a -G plugdev $USER
10. exec su-l $USER #refresh group membership
11. sudo Leap

Assuming that /usr/local/bin/ is in your path- you should just be able to run the Leap command in your shell. You need the Leap application running in the background for any other interactions with the leap device to work. Notice that I had to run the command as root even though I copied in the udev rules. In my case, running LeapPipeline(ostensibly the command line alternative to Leap) without root privileges will give an error like: [00:54:08] [Critical] [EC_NO_DEVICE] Please make sure your Leap device is plugged in.
Running LeapPipeline with root privileges does not give me any error message, just some debug output and then it quits without doing anything. Running Leap without without root privileges in my case will not give any error messages- everything will simply fail silently, i.e. if you run the visualizer you will get no tracking data.
12. Visualizer
    Test the cool new gesture features with the visualizer by running the Visualizer command and pressing the letter 'o' to enable drawing gestures. In this screenshot I have traced out a counter-clockwise circle with my finger- which is one of the supported gestures so it is drawn as light blue circular arrow. You can do multi-finger swipes, taps, and circles. See this part of the documentation for more details:

Play with the samples

At the top level of the directory where you unpacked Leap_Developer_Kit_0.7.6_3012_Linux.tar.gz you will see an examples directory. Here is what they look like.

FingerVisualizer

MotionVisualizer

UnitySandbox

Some caveats

At some point a .Leap Motion directory needs to be created in your home directory (note the space between the two words and the leading "."). I'm not sure if this is done by the installer or by the first run of the Leap application. In my case I actually copied the contents of the .Leap Motion folder on my ubuntu system to my gentoo system. In particular, there should be two files in this folder a leapid file and a file that is named something like DEV-12345 (yours will be different). The Leap application is actually a GUI app which resides in your dock/panel, if you use one, but it's not necessary- Leap will run without a panel. I've confirmed that it works with xfce4-panel if you want to see it, but you can access almost all the executables directly from the commandline anyway:

• /usr/local/bin/Visualizer
• /usr/local/bin/ScreenLocator
• /usr/local/bin/Recalibrate
• /usr/local/bin/LeapPipeline
• /usr/local/bin/Leap

Post questions if you have any problems.

The magic keyboard

As I hinted at in an earlier post, the magic trackpad hardware is very well designed and bears a strong resemblance to the Fingerworks series of input devices. Now, I've decided to extend the functionality of this device by building a wireless multitouch keyboard using two magic trackpads. Currently, this keyboard will only work in linux since it relies on my extensive modifications to the linux kernel driver for the magic trackpad.

The first image above shows my two magic trackpads with plastic overlays to indicate key placement. It turns out the magic trackpad will still detect contact through a thin insulator placed on top of its surface. The overlays are simply standard laser-printable overhead projector transparencies. I originally considered laser etching the surface of the trackpads at TechShop, but I think I will hold off until I've settled on a key arrangement that I like. Here's a picture showing how I originally planned to lay out the keys relative to my hand.



For now, I opted against the layout above in order to simplify the code that converts coordinates to keycodes. The layout I'm currently using consists simply of three concentric circles, so the code simply checks which circles contain the current touch to determine the corresponding "row" and then compares the X coordinate to a table of thresholds to determine the "column". I created the layout seen in the other pictures using Illustrator. I'll post links to PDFs for the left and right halves in case anyone feels like trying this out.

The keyboard was intended to be placed on the lap to minimize elbow flexion. To hold the halves together I used two strips of rubberized truck tarp (very strong stuff used for a windsurfing roof rack). I'm still working out how to make this attachment adjustable without being bulky.


The transparent blobs seen on the home row of the keyboard in the previous images is clear nail polish. A few coats of this allowed creating a raised ridge to facilitate finding the home row while touch typing. This is actually the principal drawback of a multitouch keyboard- the inherent lack of tactile feedback. The nail polish helps somewhat, but I'm still thinking of other ways to mitigate this issue. The image below shows the nail polish blobs close up.



I currently have preliminary mixed typing/mousing support and a few simple gestures. I also implemented certain behaviors which I think make the use of a touchpad far more ergonomic:
(1)Moving the cursor and single clicking is done with two fingers rather than one.
This helps eliminate the annoyance of inadvertent cursor movement or clicks when a single finger alights on the trackpad. I also think it's more ergonomic, since for me using a single finger requires more effort than using two.
(2)Double clicking is done with a three finger tap- this eliminates the hassle of properly timing a double click.
(3)Scrolling is done with four fingers instead of two. This allows just plopping your entire hand down on the trackpad to scroll rather than contorting it to extend only two fingers.

Todo list:

Software:
Add support for modifiers (i.e. shift,ctrl,alt,meta) using chords.

Write a GUI to rearrange key placement which generates suitable header files based on user's choices. Perhaps eventually come up with a means to allow reconfiguration on the fly.

Hardware:
Come up with better attachment and adjustment system for the straps which connect the two halves of the keyboard

To give some context for this project- here's some images of the devices the magic trackpad derives from - the iGesture and touchstream. You can see that the magic trackpad is significantly smaller than both the iGesture and touchstream which is why I think that modifiers will have to be handled only via chords (no room for separate keys). In the images below there's a weird blob on the USB cords of the iGesture and touchstream- those are my improvised hot-glue strain reliefs.

Magic Trackpad using ten fingers and with gentoo linux support

I picked up a magic trackpad this past friday and decided to see what it was capable of under linux. The video above shows that the magic trackpad hardware provides a great deal of information to the host operating system. As I'd hoped it seems to be nearly as capable as a fingerworks igesture or touchpad in some regards. In the video you can see that the trackpad is able to: detect 10 fingers, track finger contact elliptical size (i.e. along two axes) and orientation, do all the above smoothly at a high sample rate.

The video was created by streaming the debugfs file entry corresponding to the trackpad into a pygame application. The pygame code parses the apple protocol packets to determine ellipse sizes, positions and orientations and then blits them to the screen on transparent surfaces. This code is by no means pretty, in fact "quick hack" is probably a better description. Any suggestions, enhancements, criticisms,etc. are welcome.

If you want to try the code below yourself, you first need to make sure you have debugfs set up properly. Assuming you enabled debugfs when building your kernel, make sure debugfs is mounted and if it's not, mount it with something like:
mount -t debugfs none /sys/kernel/debug/
Now, if you look in /sys/kernel/debug/hid/ you should see a directory whose name corresponds to the address of your magic trackpad. If you switch into this directory you'll see an events file which you can then read with cat, tail, or the python script below. Note that support for the magic trackpad is still in a very preliminary stage, so getting things working is fairly involved. I'll describe how to play around with the hardware at a low level in the description below.

First, start with a kernel new enough to have magic-mouse support. In my case, I chose gentoo-2.6.35-r5. Now grab the multitouch branch of Chase Douglas's debian git repository. You'll need to copy or merge several files from Chase's kernel source to your own:
/drivers/hid/hid-magicmouse.c
/drivers/hid/hid-core.c
/drivers/hid/hid-ids.h
possibly a few others

If you feel like using the same kernel as Chase, just skip straight to compiling his source tree. Once the kernel is built and you've booted into it, it's time to play around with the trackpad.

Depending on the version of bluez you're using, the procedure for pairing to an HID device will vary. In bluez 3.32 you would first set the trackpad as a trusted device using DBUS, which you could either do programmatically or using a graphical tool like d-feet to call the method
/org/bluez/hci0/org.bluez.Adapter.SetTrusted("bluetooth address of your trackpad").

Once the device is set as trusted you would need to actually setup a pairing as root:
passkey-agent --default 0000
then use d-feet or DBUS CLI to call CreateDevice("BluetoothAddress"). At this point if you call ListDevices() you should a new device corresponding to the trackpad.
Once you've paired with the trackpad and loaded the hid-magicmouse kernel module you should see messages in the system log indicating that a new input device has been registered.

input: Apple Wireless Trackpad as /class/input/input6
magicmouse 0005:05AC:030E.0005: input,hidraw4: BLUETOOTH HID v1.60 Mouse [Apple Wireless Trackpad] on 00:27:48:09:63:60
input: Apple Wireless Trackpad as /class/input/input7


Note that you'll also want an up to date installation of evdev, otherwise you might get messages like this:

evdev.c(EVIOCGBIT): Suspicious buffer size 511, limiting output to 64 bytes. See http://userweb.kernel.org/~dtor/eviocgbit-bug.html


If you want to see the touch reports the trackpad produces you can use the evtest application. In gentoo this is provided by the joystick ebuild. Now if you do

]$ evtest /dev/input/event6
Input driver version is 1.0.0
Input device ID: bus 0x5 vendor 0x5ac product 0x30e version 0x160
Input device name: "Apple Wireless Trackpad"
Supported events:
Event type 0 (Sync)
Event type 1 (Key)
Event code 272 (LeftBtn)
Event code 325 (ToolFinger)
Event code 330 (Touch)
Event code 333 (Tool Doubletap)
Event code 334 (Tool Tripletap)
Event code 335 (?)
Event type 3 (Absolute)
Event code 0 (X)
Value 3097
Min -2909
Max 3167
Event code 1 (Y)
Value 2238
Min -2456
Max 2565
Event code 48 (?)
Value 0
Min 0
Max 255
Event code 49 (?)
Value 0
Min 0
Max 255
Event code 52 (?)
Value 0
Min -32
Max 31
Event code 53 (?)
Value 0
Min -2909
Max 3167
Event code 54 (?)
Value 0
Min -2456
Max 2565
Event code 57 (?)
Value 0
Min 0
Max 15
Event type 4 (Misc)
Event code 3 (RawData)
Testing ... (interrupt to exit)


Now if you touch with two fingers you'll see a slew of output which should include something like:

Event: time 1283194689.870573, -------------- Config Sync ------------
Event: time 1283194689.870586, type 1 (Key), code 330 (Touch), value 0
Event: time 1283194689.870587, type 1 (Key), code 333 (Tool Doubletap), value 0
Event: time 1283194689.870589, -------------- Report Sync ------------


Fingerworks iGesture

Imagine half of a touchstream, without the typing capability, and this is what you get. Just like the touchstream, the USB cord is not detachable and poses a potential point of failure. The picture above shows an improvised strain relief on the cord using hot glue.

Touchstream LP

Amazing, but has a very high learning curve and an even higher price ( if you can even find one). I generally use mine when I'm doing work that involves more complicated mouse actions than actual typing or when browsing the web (since I have all the browser actions mapped to gestures).

Pros:
Combined typing and mousing. Being able to seamlessly transition from typing to mousing or gesturing without moving your hands is great.

Highly programmable- literally hundreds of user configurable gestures and settings. Imagine bettertouchtool on steroids x 2.

Cons:
Lack of tactile feedback makes typos a constant issue. Even after owning several touchstream keyboards for many years, I still type much more slowly on these than on a mechanical keyboard and make far more typos.

Availability- the only way you might be able to get ahold of one now is ebay.

Price- absurdly high due to the lack of supply.