Tuesday 7 February 2012

Tutorial 3: Getting the Video Stream

The last tutorial was a bit boring, all we did was move the sensor up and down, this time we are going to get the RGB Video Stream from the Kinect Sensor and display it in our window.

Up to thirty times a second the Kinect Sensor will get an image from the RGB camera and to display this as a video stream so first we need to add a texture to store our color video frame so we will add this just below our Kinect Sensor like so:
KinectSensor kinect;
Texture2D colorVideo;
Now we have a Texture2D object to store our images in we need to actually tell the Kinect Sensor to enable the RGB Video Stream and what to do every time we get a frame from the camera. In the Initialize method we need to add the following lines:
kinect.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
kinect.ColorFrameReady += new EventHandler<ColorImageFrameReadyEventArgs>(kinect_ColorFrameReady);
These go after we get the Kinect Sensor and before we Start the Kinect Sensor, you should now have something that is similar to this:
kinect = KinectSensor.KinectSensors[0];
kinect.ColorStream.Enable(ColorImageFormat.RgbResolution640x480Fps30);
kinect.ColorFrameReady += new EventHandler<ColorImageFrameReadyEventArgs>(kinect_ColorFrameReady);
kinect.Start();
What we have done here is enabled the Color Stream for our Kinect Sensor with the Image Format being RGB, 640 pixels wide and 480 pixels high at 30 FPS.

We then tell the Kinect Sensor what to do each time we receive a Color Frame from the Kinect Sensor, in this instance by adding a new event handler to the sensor to call the method kinect_ColorFrameReady. 

Color Frame Ready Event Handler
Now we have told the Kinect Sensor to call a new method we should probably create this. I personally like to keep my code tidy so I have added this new method below the draw method, rather than mixing it with the standard XNA methods.

To start off we need to create the method called kinectColorFrameReady:
void kinect_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs colorImageFrame)
{
    // Open image
    ColorImageFrame colorVideoFrame = colorImageFrame.OpenColorImageFrame();
}
Every time we get a frame from the Kinect Sensor it will pass through an Object that fire the event (sender) and a ColorImageFrameReadyEventArgs object containing an image we open into our ColorImageFrame variable. Next we need to get the Pixel Data from the ColorImageFrame so we  add the following to our method:
if (colorVideoFrame != null)
{
    //Create array for pixel data and copy it from the image frame
    Byte[] pixelData = new Byte[colorVideoFrame.PixelDataLength];
    colorVideoFrame.CopyPixelDataTo(pixelData);
}
Here we first check if our ColorImageFrame is null, when the project is still initialising the Kinect Sensor is already sending data but it may not contain anything!  Next we create a Byte array that is the same length as the Pixel Data, we then copy the Pixel Data to our byte array.

Create The Video Texture
Now we have our Pixel Data we need to create a texture from this for us to display.  Earlier we created a Texture2D variable that we will now initialise with the following line just below copying the pixel data.
colorVideo = new Texture2D(graphics.GraphicsDevice, colorVideoFrame.Width, colorVideoFrame.Height);
Now we need to pass our Pixel Data to the Texture2D variable using the following line:
colorVideo.SetData(pixelData);
Draw The Video Texture 
Now we have the video texture being updated for each frame we need to draw this on the screen so we can see the series of images as a video stream.  Update your Draw method to look like this:
protected override void Draw(GameTime gameTime)
{
    GraphicsDevice.Clear(Color.CornflowerBlue);
    
    // Draw RGB video
    spriteBatch.Begin();
    spriteBatch.Draw(colorVideo, new Rectangle(0, 0, 640, 480), Color.White);
    spriteBatch.End();

    base.Draw(gameTime);
}
Success?
As you can probably see our video appears to have a blue tint, this is because the camera is capturing the video in RGBA and annoyingly XNA uses BGRA, I am not sure why Microsoft decided to use BGRA but all we need to do is re-align the byes before we set our pixel data.

Removing The Blue Tint
It is quite simple to fix this issue as the Kinect Sensor is passing 0 0 255 0 for Blue but XNA is displaying this as BGRA so what we actually get is Red.
Go back to the kinect_ColorFrameReady method we created and place the following code after copying the PixelData and before creating the texture:
//Convert RGBA to BGRA
Byte[] bgraPixelData = new Byte[colorVideoFrame.PixelDataLength];
for (int i = 0; i < pixelData.Length; i += 4)
{
    bgraPixelData[i] = pixelData[i + 2];
    bgraPixelData[i + 1] = pixelData[i + 1];
    bgraPixelData[i + 2] = pixelData[i];
    bgraPixelData[i + 3] = (Byte)255; //The video comes with 0 alpha so it is transparent
}
You will notice that we are setting the fourth byte to 255, this is because the RGBA sent from the camera has an Alpha value of 0 making it transparent, setting this to 255 makes the image solid so it does not blend with the CornflowerBlue background. Now we need to update our setData call to use bgraPixelData like so:
colorVideo.SetData(bgraPixelData);
Your finished kinect_ColorFrameReady method should look something like this:
void kinect_ColorFrameReady(object sender, ColorImageFrameReadyEventArgs colorImageFrame)
{
    //Get raw image
    ColorImageFrame colorVideoFrame = colorImageFrame.OpenColorImageFrame();

    if (colorVideoFrame != null)
    {
        //Create array for pixel data and copy it from the image frame
        Byte[] pixelData = new Byte[colorVideoFrame.PixelDataLength];
        colorVideoFrame.CopyPixelDataTo(pixelData);

        //Convert RGBA to BGRA
        Byte[] bgraPixelData = new Byte[colorVideoFrame.PixelDataLength];
        for (int i = 0; i < pixelData.Length; i += 4)
        {
            bgraPixelData[i] = pixelData[i + 2];
            bgraPixelData[i + 1] = pixelData[i + 1];
            bgraPixelData[i + 2] = pixelData[i];
            bgraPixelData[i + 3] = (Byte)255; //The video comes with 0 alpha so it is transparent
        }

        // Create a texture and assign the realigned pixels
        colorVideo = new Texture2D(graphics.GraphicsDevice, colorVideoFrame.Width, colorVideoFrame.Height);
        colorVideo.SetData(bgraPixelData);
    }
}
The result:
 
Wrapping Up
In this tutorial we covered setting up the Kinect Sensor to use a Color Video Stream, adding an event handler to the Kinect Sensor that is fired every time it reads a Color Video Frame and fixing the pixel data to display the image correctly in our window. Finally we have something to show from the Kinect Sensor!  Next time we will do the same with the Depth Video Stream.

Solution: Tutorial 3 Solution

16 comments:

  1. Looking super happy there Nick!

    ReplyDelete
  2. It is very hard to look happy without looking creepy! I will probably change the picture at some point.

    ReplyDelete
  3. Loving these tutorials...I spent half the day looking for some that would be up to date with the latest SDK but nothing, till now of cause :D
    Any chance of sharing the source code, as i'm having issues

    ReplyDelete
  4. Hi RStaff,

    I most certainly am happy to share the full source code. I have been meaning to upload it somewhere but have not had the time recently.

    I will get the source code up for each tutorial shortly after I publish the Skeletal Tracking which will hopefully be at the weekend.

    What issues are you having?

    ReplyDelete
    Replies
    1. Hi Nick,

      The first issue I come across is two errors that occur when creating the event handler....
      1:No overload for 'kinect_ColorFrameReady' matches delegate
      2: The type or namespace name 'colorimageframereadyeventargs' could not be found.

      is there something im missing or am i just being an idiot?

      Delete
    2. Hi RStaff,

      I noticed the SyntaxHighlighter did mess up my snippets because I left the < and > in. I have now tidied these up; this will fix issue 2.

      Issue 1 could be related as the event handler would not have been defined properly. If you adjust these two bits it should work.

      I have also popped the Solution into my public Dropbox folder so I have put the link ^^ (still sorting out my own hosting)

      Delete
    3. Thanks for your reply, I eventually solved the problem. I typed "ColorImageFrameReadyEventArgs" incorrectly, I didnt put capitals where needed...silly mistake.

      thanks for posting the solution too

      Delete
  5. Hey, great tutorials by the way! I just have one minor problem, my debug panel keeps printing "Warning: a ImageFrame instance was not disposed", is there any way to fix this or have I just missed something?

    ReplyDelete
    Replies
    1. I Imagine it will be a difference between the V1.0 SDK and the latest version. I am yet try the new SDK.

      Delete
  6. Hello Nick, thank you for your tutorial, but I have a problem with EventHandler(kinect_ColorFrameReady);

    visual studio says: Error 1 The name 'Kinect_ColorFrameReady' does not exist in the current context

    do you know how to solve it?

    thanks in advance

    Márcio

    ReplyDelete
    Replies
    1. Hi Márcio,

      I think this will be related to the SDK versions. I originally wrote these for V1.0 and now I think they have V1.8.

      I assume the comment above does not help with your issue?

      Delete
    2. when your done using the frame that you processed simply call frame.Dispose()

      Delete
  7. Hi there, I know this is a little old now, but I hope you're still tracking it. The issue I am encountering is that the kinect_ColorFrameReady method is never being run. Which means that when I hit the draw Method, the texture is returning null and throwing an error.

    On a different note, however. You are a hero, these tutorials are an oasis in a desert of indecypherable techno-speak (I am relatively new to programming, though I have made games in XNA before). So... thanks!

    ReplyDelete
  8. Nevermind, I commented prematurely. It was all my mistake. Sorry for wasting your time, and thanks again!

    ReplyDelete
  9. Hi when i try to run the above code i get kinect_ColorFrameReady does not exist in the current context... can anyone please help me?

    ReplyDelete
  10. Great!!! Good tutorial and you save me with the tint blue, I use in Intel Real Sense camera and xna :D
    And other solution in Kinect v2 you can use
    ...new Texture2D(graphicsDeviceManager.GraphicsDevice, colorFrameDescriptionWithHeigth.Width, colorFrameDescriptionWithHeigth.Height, false, SurfaceFormat.Color);

    and then

    if ((colorFrameDescription.Width == colorFrameDescriptionWithHeigth.Width) && (colorFrameDescription.Height == colorFrameDescriptionWithHeigth.Height))
    // {
    if (colorFrame.RawColorImageFormat == ColorImageFormat.Rgba)
    {
    colorFrame.CopyRawFrameDataToArray(this.colorPixels);
    // }
    else
    {
    colorFrame.CopyConvertedFrameDataToArray(this.colorPixels, ColorImageFormat.Rgba);
    }

    colorFrameProcessed = true;
    }
    this.imagenCamaraKinect.SetData(colorPixels);

    ReplyDelete