Friday, December 17, 2010

HOWTO: use the kinect as a mouse in linux



In an earlier post I explained how to get PrimeSense's NITE up and running and how to use the samples they provided. Now some people might be thinking "cool, but how can I use this?" I thought using NITE hand tracking to control the cursor would be a good and simple demonstration.

The linux kernel provides a means to create userspace input drivers using a feature called uinput. If you compile your kernel with uinput enabled as a module you can then simply:
modprobe uinput
to load the uinput module. Once the module is loaded you can use the piece of code I've embedded below to convert the coordinates output by the NITE code into actual mouse/cursor movement. In short:

(1) download the code below
(2) save it as ~/kinect/NITE/Nite-1.3.0.17/Samples/SingleControl/main.cpp (you might want to back up the original)
(3)cd ~/kinect/NITE/Nite-1.3.0.17 && make
(4)Note: do the following as root or using sudo

~/kinect/NITE/Nite-1.3.0.17/Samples/Bin/Sample-SingleControl

(5)Perform a focus gesture to start the hand tracking (check out my video above to see how to do that)

At this point you should be able to do what I do in the video above. You can also extend the code to generate mouse clicks, keystrokes, etc. Have fun.
At some point dropbox ate the public link for the source code I was using before and in the process of restoring from an old backup the formatting of the code below got a bit mangled... in any case here's a gist of what I salvaged:

19 comments:

  1. Hi,

    I think I've ported the essence of your implementation on Windows without synergy.

    ~~~~~~~~~~~~~~~~

    #include conio.h
    #define WIN32_LEAN_AND_MEAN // Exclude Extra Windows Crap
    #define WIN32_EXTRA_LEAN // Exclude More Windows Crap
    #include windows.h

    ~~~~~~~~~

    void XN_CALLBACK_TYPE OnPointUpdate(const XnVHandPointContext* pContext, void* cxt)
    {
    //printf("%d: (%f,%f,%f) [%f]\n", pContext->nID, pContext->ptPosition.X, pContext->ptPosition.Y, pContext->ptPosition.Z, pContext->fTime);
    printf("%d: (%f,%f,%f) [%f]\n", pContext->nID, pContext->ptPosition.X*2.5, pContext->ptPosition.Y*2.5, pContext->ptPosition.Z, pContext->fTime);
    SetCursorPos((int)(pContext->ptPosition.X*2.5),(int) -(pContext->ptPosition.Y*2.5));
    }

    ~~~~~~~~~~~~~~~~~~~~~~

    Then you'd change constant 2.5 depending on resolution and sensitivity.

    Thanks for keeping your code open so other people can use it.

    ReplyDelete
  2. Hi, have you figured out by now how to do multiple hand tracking?

    ReplyDelete
  3. Firstname, yeah, I was trying to throw together some code using multipointer X to demonstrate it, but for now I'll just add a mention of the config file change that's needed and the contents of an XnHandPointContext.

    ReplyDelete
  4. Hello,

    Is there any way that works on windows?

    ReplyDelete
  5. Hey trtg,

    Do you have a public email address for this blog that I could reach you at?

    ReplyDelete
  6. Jared, try the "Contact me" form at the bottom of each page. Please be aware,however, that if you have some lengthy support request, I probably won't be able to help you. I do not work for Primesense or any other related company and only have limited time to dedicate to this blog.

    ReplyDelete
  7. when i run (4)~/kinect/NITE/Nite-1.3.0.17/Samples/Bin/Sample-SingleControl

    it says illegal construction. whats wrong with it?

    ReplyDelete
  8. For the illegal instruction issue, try running the program under gdb to locate the source of the illegal instruction call.

    If it comes from libXnVFeatures or libXnVHandGenerator try downloading the newest unstable Nite build from http://www.openni.org/downloadfiles/openni-compliant-middleware-binaries/33-latest-unstable.

    I had an issue regarding illegal opcodes and my AMD x86_64 processor. The new Nite build seemed to resolve it.

    ReplyDelete
  9. Is there anyway I can see if I have uinput compiled in my kernel and enabled as a module on my system.
    I'm running Mint Linux 9

    Cheers!

    ReplyDelete
  10. Jeff,
    The most straightforward way would be to simply try running (as root):
    modprobe uinput

    If that command fails, uinput probably wasn't enabled in your kernel. If the output of modprobe somehow wasn't clear enough you can try running the command:
    modinfo uinput

    If the uinput module is present on your system you should see a bunch of info. More directly you can try

    lsmod | grep uinput

    to check whether the uinput module is already loaded.

    ReplyDelete
  11. It runs, outputs the hand coordinates in terminal, however mouse doesnt move. I have uinput and did modprobe, but nothing seems to happen. There are no error messages either

    ReplyDelete
  12. The problem seems to be basically the same as :
    http://ubuntuforums.org/showthread.php?t=1617581

    ReplyDelete
  13. Mike, I forgot to mention in my original post that unless you've set up permissions so that your user has full access to /dev/uinput you will need to run Sample-SingleControl as root. This may resolve your issue.

    ReplyDelete
  14. Wait is there a way to click on this?

    ReplyDelete
  15. Hi

    whenever i try run this it throw me this error Couldn't initialize: File not found!

    :( even i run it as su

    ReplyDelete
  16. @uahmed: there should be a line somewhere in the main.cpp that looks like this:

    #define SAMPLE_XML_FILE "Data/Sample-Tracking.xml"

    Make sure that Sample-Tracking.xml is in the correct place relative to the working directory, otherwise the program won't be able to set up the module.

    @trtg: trying to get clicking to work properly - I figured it would be something like:

    void click_mouse()
    {
    memset(&event, 0, sizeof(event));

    gettimeofday(&event.time, NULL);

    event.type = EV_KEY;
    event.code = BTN_MOUSE;
    event.value = BTN_LEFT;
    write(uinp_fd, &event, sizeof(event));

    event.type = EV_SYN;
    event.code = SYN_REPORT;
    event.value = 0;
    write(uinp_fd, &event, sizeof(event));
    }

    Probably going to use two hands, one for clicking and one for moving the cursor (for this I'll cannibalise the PointViewer example). Thanks for the howtos - they've been really helpful!

    ReplyDelete