Sunday 26 February 2012

Tutorial 5: Skeletal Tracking

Time for some skeleton tracking.  I know it has been a while since the last tutorial so hopefully this will keep you going.

Tracking skeletons with Kinect takes a bit more than using the Image and Depth cameras so this might be a bit longer than the other tutorials.  We need somewhere to store the Skeleton Data for all skeletons and somewhere to store our Tracked skeleton so add these variables at the top of our code:
Skeleton[] skeletonData;
Skeleton skeleton;
In the Depth Sensor tutorial we started using the AllFramesReady event handler and we will continue with this approach in this tutorial, so we need to enable the Skeleton Stream.  To enable the skeleton stream we simple add the following line with our Kinect initialisation:
kinect.SkeletonStream.Enable();
After enabling the Skeleton Stream we need to do something with the data, below the Depth Image handling in our kinect_AllFramesReady method we need to collect the skeleton data from the sensor.  First we open the Skeleton Frame and test if it is null (no skeletons being tracked)
//
// Skeleton Frame
//
using (SkeletonFrame skeletonFrame = imageFrames.OpenSkeletonFrame())
{
    if (skeletonFrame != null)
    {

    }
}
If we have no skeletons being tracked we do not want to do anything but if we have one or more skeletons that are tracked we need to get the data from the skeleton frame.

First we check is the skeletonData is empty or if the data from the previous frame has changed (more or less skeletons being tracked); if we meet one of these conditions we initialise our Skeleton Data array to the correct length.  Add the following for when the skeletonFrame is not null:
if ((skeletonData == null) || (this.skeletonData.Length != skeletonFrame.SkeletonArrayLength))
{
    this.skeletonData = new Skeleton[skeletonFrame.SkeletonArrayLength];
}
//Copy the skeleton data to our array
skeletonFrame.CopySkeletonDataTo(this.skeletonData);
We have also added the line to copy the skeleton data from our Skeleton Frame to the Skeleton Data array.

Selecting The Skeleton
After taking the skeleton data from the frame we need to actually check the skeletons for their tracking state.  Below our previous section of code we need to add this:
if (skeletonData != null)
{
    foreach (Skeleton skel in skeletonData)
    {
        if (skel.TrackingState == SkeletonTrackingState.Tracked)
        {
            skeleton = skel;
        }
    }
}

After the usual check for an empty variable we loop through each skeleton in our skeletonData array; we check the tracking state to be Tracked and if so we copy the data to our Skeleton variable.  I have kept this simple for now as this is quite a lengthy tutorial; in following tutorials I will cover two player tracking and switching active players.

The finished section of code should look like:
using (SkeletonFrame skeletonFrame = imageFrames.OpenSkeletonFrame())
{
    if (skeletonFrame != null)
    {
        if ((skeletonData == null) || (this.skeletonData.Length != skeletonFrame.SkeletonArrayLength))
        {
            this.skeletonData = new Skeleton[skeletonFrame.SkeletonArrayLength];
        }

        //Copy the skeleton data to our array
        skeletonFrame.CopySkeletonDataTo(this.skeletonData);
    }
}

if (skeletonData != null)
{
    foreach (Skeleton skel in skeletonData)
    {
        if (skel.TrackingState == SkeletonTrackingState.Tracked)
        {
            skeleton = skel;
        }
    }
}

We are now handling the Skeleton Stream by copying the data to our Skeleton Data array, then looping through each skeleton and only processing those that are tracked.

Drawing The Joints
Simply processing the data would not be a fun tutorial so we will actually draw the joints for our skeleton; again I have kept this simple for now. Let's create a new method called DrawSkeleton:
private void DrawSkeleton(SpriteBatch spriteBatch, Vector2 resolution, Texture2D img)
{
    if (skeleton != null)
    {
        foreach (Joint joint in skeleton.Joints)
        {

        }
    }
}
We are passing in the SpriteBatch we want to draw the Joints with, the Resolution of our program and a Texture for the joints.  You can see that we are going to loop through the Joints of our skeleton and process each joint.

To draw the Joints first we need to calculate their coordinates, we can do that with the following:
Vector2 position = new Vector2((((0.5f * joint.Position.X) + 0.5f) * (resolution.X)), (((-0.5f * joint.Position.Y) + 0.5f) * (resolution.Y)));
It seems very confusing at first but in reality we are doing the same thing for both axis, we are halving the position of the joint adding a 0.5f correction and repositioning it based on the resolution of our program.  This was essentially trial and error to find a method that worked.

After calculating the position we need to draw it:
spriteBatch.Draw(img, new Rectangle(Convert.ToInt32(position.X), Convert.ToInt32(position.Y), 10, 10), Color.Red);

This will draw a circle at our calculated position giving us this:
As you can see a representation of a skeleton without bones.  The complete Draw Skeleton method:
private void DrawSkeleton(SpriteBatch spriteBatch, Vector2 resolution, Texture2D img)
{
    if (skeleton != null)
    {
        foreach (Joint joint in skeleton.Joints)
        {
            Vector2 position = new Vector2((((0.5f * joint.Position.X) + 0.5f) * (resolution.X)), (((-0.5f * joint.Position.Y) + 0.5f) * (resolution.Y)));
            spriteBatch.Draw(img, new Rectangle(Convert.ToInt32(position.X), Convert.ToInt32(position.Y), 10, 10), Color.Red);
        }
    }
}

Wrapping Up
In this tutorial we have covered enabling the Skeleton Stream, copying the skeleton data from to our array, processing the skeletons and then drawing each point.  I have kept this tutorial relatively short and simple as there is quite a lot going on behind the scenes and we will cover this in more detail in following tutorials, such as mapping the joints to the depth image, drawing bones and multiple player tracking.

Solution: Tutorial 5 Solution

21 comments:

  1. Scoooore...ive been waiting on this all weekend, cant seem to get my own one working lol
    Cheers for this keep em coming :D

    ReplyDelete
    Replies
    1. Have you had any luck getting your own one working?

      Started writing the next tutorial although they are getting more complex so there may be longer gaps between tutorials.

      Delete
    2. Hey, yeh I managed to get both this and the previous one up and running in the end :D
      Think I'll study them both more and eventually learn to use the body movements

      Delete
    3. im working on rigging a 3D character using kinect data, i have been able to get the quaternion rotations working but i can't get the character to move (translate), any ideas/quick suggestions?

      Delete
    4. Odd that you cant get it to translate...do you want this to happen when the player moves (translates) or a way of the player making the rigged 3D character move (translate) ?
      If the latter, then could you not store various bone positions and rotations in a data structure like an array or list and constantly run a check for those positions on the players skeleton, when the player matches those positions, the rig runs or translates forward etc?

      Delete
    5. I follow this site with great interest. I have made up a fully rigged 3d character in 3ds Max and was successful to get the character and animations over into a XNA game environment. The next logical step would be using Kinect and take the skeleton data and copy the information over to the bone structure of the 3d character. I really look forward to seeing an algorithm how to do that, bringing “live” animation from Kinect to a XNA character. Keep up the good work, Nick!

      Delete
  2. estas bien chingones los tutoriales pero ya saca uno porfa que nos quedamos picados.Seria bueno que sacaras uno donde sincronices un modelo 3d con la entrada del kinect.

    ReplyDelete
  3. Replies
    1. Unfortunately not yet, I have become super busy and have not had many opportunities to sit down and code. Hopefully I will get some time soon.

      Delete
  4. Is there a way to map a desperate image to each point??

    ReplyDelete
    Replies
    1. Is there a way to map a different image to each point??

      Delete
    2. Yes there is, where there is this call:

      spriteBatch.Draw(img, new Rectangle(Convert.ToInt32(position.X), Convert.ToInt32(position.Y), 10, 10), Color.Red);

      The img parameter is the texture we are using. You could easily change this img by checking the joint and matching it to a specific JointType and using an enum for the images.

      Hopefully this is not too vague. I will try and do an example when I get time.

      Nick

      Delete
  5. i followed yo instructions and did all 5, thanx a bunch man :D
    is there a way to map like a shirt, u knw 2 or more positions to an image :)

    ReplyDelete
  6. do u have sample code for virtual dress room , if so can you please share that too.

    Regards.
    jay

    ReplyDelete
  7. Need a little help with this tutorial here....i cannot get the program to draw the joints (like the test result image above) can anyone help me?? i'm new at this.. thank you...

    ReplyDelete
    Replies
    1. Hi, are you getting any errors in the console?

      Delete
    2. nope...no error at all...

      Delete
  8. Getting an error in the previous tutorial. Someone with the same error left a post with no reply, any help would be greatly appreciated.

    ReplyDelete
    Replies
    1. What SDK version are you using? I can see that 1.6 is the latest and these tutorials were originally written for 1.0.

      At a guess some of the methods may have changed causing some issues.

      Delete
  9. Many thanks, your tutorial was a great help in getting my group started on its Kinect-enabled application.

    ReplyDelete
  10. are you not going to make some tutorial about recognizing gesture?

    ReplyDelete