Getting Started with the Kinect

We got a Kinect in the lab this week and so I restructured my schedule a little bit in order to get my hands on it as quickly as possible.

I've got my laptop set up with the LibFreenect open source library for interfacing with the Kinect device, and begun to extract some data:

These images are screen grabs from one of the LibFreenect sample projects and show the extracted RGB image from the Kinect and a depth image that has been colorized (red is close to the camera dark blue is far away, black indicates a lack of depth information).

From this output, I've determined a couple of important details about the Kinect. The user has to be about two feet away from the Kinect or else the device will not be able to accurately determine their depth. This occurs because the Kinect uses an infrared painter to cast rays into the environment which the camera picks up, and objects that are too close to the Kinect get hit with too many rays that they get washed out. There is also a depth "shadow" that gets cast such that when one object is in front of another, the device cannot determine the depth of the background object around the left border of the foreground object.

I don't think that there should be any significant future problem based on these artifacts because I only need to track the hands with precision (and they should be in front of everything), and get a general positioning of the head (I don't need facial features or any specific data, just a generalized head position) so even if the player's hands are obscuring their face I shouldn't have trouble gathering the information I need.

It seems like working with the LibFreenect library moving forward should be fairly straight forward. The library provides a series of c-style function calls to the Kinect device, and I think that wrapping a class around these should allow me to integrate the library into my application in a suitably abstract fashion.

Next week I'll be researching the hand/head recognition algorithms/software that I want to use, and getting something resembling the above demo working using DirectX for rendering.

Senior Project Kickoff Post

This blog will be detailing the progress of my senior design capstone project. I will be completing the project over the course of the spring 2011 semester.

My project is the creation of virtual building block experience utilzing the Microsoft Kinect hardware system; here is the abstract: 

I plan to simulate a building block experience using the Microsoft Kinect hardware system. I will use the Kinect to track the player’s hands and head in order to present them with objects that appear to project out of their screens that they can pick up and move around as if they were real. The objects will be given real world physical properties to create a realistic building block experience, where unstable structures crumble, and improperly placed pieces can bring the entire structure crashing down. The objects will also be given interesting shapes to encourage user creativity and experimentation. The project will use the Libfreenect open source library to extract information from and send orders to the Kinect device and graphics output will be handled by the Microsoft DirectX libraries.

My anticipated work for next week:

1. Complete background research
1. Research face/head recognition algorithms
2. Research hand recognition algorithms
3. Determine whether to use existing physics system or roll my own 
2. Begin framework setup integrating DirectX at least
3. Figure out how I'm going to obtain a Kinect :)