Kinect for Windows in Processing 3

Posted by bryan

Finally, the skeleton part of the library is done. In this very experimental version, I extract only one skeleton and store the joints information in an array (size 20) of PVector type in Processing. The tracking state is not implemented yet. I use the z-depth value to indicate if that joint is validly tracked or not. The x and y values are normalized to the range from 0 to 1 in the screen space. In the next version, I would like to implement the async event of tracking. It is time to integrate the all three components:

  1. Individual RGB and depth images
  2. Aligned RGB and depth image
  3. Skeleton tracking

 

 
Here is a copy of the sample Processing code.

import pKinect.PKinect;
 
PKinect kinect;
PVector [] loc;
 
void setup()
{
  size(640, 480);
  kinect = new PKinect(this);
  smooth();
  noFill();
  stroke(255, 255, 0);
}
 
void draw()
{
  background(0);
  loc = kinect.getSkeleton();
  drawSkeleton();
}
 
void drawSkeleton()
{
  // Body
  DrawBone(kinect.NUI_SKELETON_POSITION_HEAD,
  kinect.NUI_SKELETON_POSITION_SHOULDER_CENTER);
  DrawBone(kinect.NUI_SKELETON_POSITION_SHOULDER_CENTER,
  kinect.NUI_SKELETON_POSITION_SHOULDER_LEFT);
  DrawBone(kinect.NUI_SKELETON_POSITION_SHOULDER_CENTER,
  kinect.NUI_SKELETON_POSITION_SHOULDER_RIGHT);
  DrawBone(kinect.NUI_SKELETON_POSITION_SHOULDER_CENTER,
  kinect.NUI_SKELETON_POSITION_SPINE);
  DrawBone(kinect.NUI_SKELETON_POSITION_SPINE,
  kinect.NUI_SKELETON_POSITION_HIP_CENTER);
  DrawBone(kinect.NUI_SKELETON_POSITION_HIP_CENTER,
  kinect.NUI_SKELETON_POSITION_HIP_LEFT);
  DrawBone(kinect.NUI_SKELETON_POSITION_HIP_CENTER,
  kinect.NUI_SKELETON_POSITION_HIP_RIGHT);
 
  // Left Arm
  DrawBone(kinect.NUI_SKELETON_POSITION_SHOULDER_LEFT,
  kinect.NUI_SKELETON_POSITION_ELBOW_LEFT);
  DrawBone(kinect.NUI_SKELETON_POSITION_ELBOW_LEFT,
  kinect.NUI_SKELETON_POSITION_WRIST_LEFT);
  DrawBone(kinect.NUI_SKELETON_POSITION_WRIST_LEFT,
  kinect.NUI_SKELETON_POSITION_HAND_LEFT);
 
  // Right Arm
  DrawBone(kinect.NUI_SKELETON_POSITION_SHOULDER_RIGHT,
  kinect.NUI_SKELETON_POSITION_ELBOW_RIGHT);
  DrawBone(kinect.NUI_SKELETON_POSITION_ELBOW_RIGHT,
  kinect.NUI_SKELETON_POSITION_WRIST_RIGHT);
  DrawBone(kinect.NUI_SKELETON_POSITION_WRIST_RIGHT,
  kinect.NUI_SKELETON_POSITION_HAND_RIGHT);
 
  // Left Leg
  DrawBone(kinect.NUI_SKELETON_POSITION_HIP_LEFT,
  kinect.NUI_SKELETON_POSITION_KNEE_LEFT);
  DrawBone(kinect.NUI_SKELETON_POSITION_KNEE_LEFT,
  kinect.NUI_SKELETON_POSITION_ANKLE_LEFT);
  DrawBone(kinect.NUI_SKELETON_POSITION_ANKLE_LEFT,
  kinect.NUI_SKELETON_POSITION_FOOT_LEFT);
 
  // Right Leg
  DrawBone(kinect.NUI_SKELETON_POSITION_HIP_RIGHT,
  kinect.NUI_SKELETON_POSITION_KNEE_RIGHT);
  DrawBone(kinect.NUI_SKELETON_POSITION_KNEE_RIGHT,
  kinect.NUI_SKELETON_POSITION_ANKLE_RIGHT);
  DrawBone(kinect.NUI_SKELETON_POSITION_ANKLE_RIGHT,
  kinect.NUI_SKELETON_POSITION_FOOT_RIGHT);
}
 
void DrawBone(int _s, int _e)
{
  if (loc == null)
  {
    return;
  }
  PVector p1 = loc[_s];
  PVector p2 = loc[_e];
  if (p1.z == 0.0 || p2.z == 0.0)
  {
    return;
  }
  line(p1.x*width, p1.y*height, p2.x*width, p2.y*height);
}

This version of the code (for Windows 7, both 32-bit and 64-bit) is available here.

Kinect for Windows in Processing 2

Posted by bryan

After the first post of the Kinect for Windows library, this is the second version of the Kinect for Windows in Processing. I modified the pixels array to contain only the colour pixels of players. The rest is painted green as background. In this version, I align the colour and depth pixels in one output frame buffer, with the use of the NuiImageGetColorPixelCoordinateFrameFromDepthPixelFrameAtResolution method.
 

 
The Processing code is here. The DLL is in the code folder of the sketch. In this version, I used Windows 7 32-bit and Java 7. I’ll consolidate to build both the 32-bit and 64-bit once the interface is more stable.

I use the Kinect for Windows 1.5 SDK.

Kinect for Windows in Processing 1

Posted by bryan

Finally, I start to work on a Processing library to work with Kinect for Windows SDK. This very preliminary version shows only the colour image and the depth image. For the depth image, I convert the original 13 bit depth map into a PImage for easy display. Both images are of the size 640 x 480.

You can download this for a try. It is created in Windows 7 64bit and Java 7. More to work!

The code is very straightforward.

import pKinect.PKinect;
 
PKinect kinect;
 
void setup()
{
  size(1280, 480);
  background(0);
  kinect = new PKinect(this);
}
 
void draw()
{
  image(kinect.GetImage(), 0, 0);
  image(kinect.GetDepth(), 640, 0);
}
 
void mousePressed()
{
  println(frameRate);
}

Kinect for Windows

Posted by bryan

The new Kinect for Windows is available and will be available in Hong Kong in late May according to the blog description. The commercial SDK is also out now for download.

TouchDesigner has a set of new operators for the Kinect for Windows. They are quite easy to integrate with the existing TouchDesigner working environment.

University of Central Florida Interactive Systems and User Experience Lab has also released a Unity3D plugin with the new Kinect SDK.
 

 
There are also a number of openFrameworks addons for the Kinect for Windows, ofxMSKinect and ofxKinectNui.

For library Cinder, here is the Kinect SDK Block.

For Flash ActionScript users, the AIRKinect can be a good choice.

At the time of writing, I am still waiting for the Java binding and thus the Processing community.