EDN Admin
Well-known member
This video covers the basics of reading camera data from the Kinect sensor. You may find it easier to follow along by downloading the http://files.ch9.ms/coding4fun/KinectForWindowsSDKQuickstarts.zip Kinect for Windows SDK Quickstarts samples and slides . [ http://channel9.msdn.com/Series/KinectSDKQuickstarts/Camera-Fundamentals#time=0m25s 00:25 ] Camera data information [ http://channel9.msdn.com/Series/KinectSDKQuickstarts/Camera-Fundamentals#time=3m30s 03:30 ] Creating the UI [ http://channel9.msdn.com/Series/KinectSDKQuickstarts/Camera-Fundamentals#time=4m48s 04:48 ] Initializing the Kinect runtime [ http://channel9.msdn.com/Series/KinectSDKQuickstarts/Camera-Fundamentals#time=7m18s 07:18 ] Reading values from the RGB camera [ http://channel9.msdn.com/Series/KinectSDKQuickstarts/Camera-Fundamentals#time=11m26s 11:26 ] Reading values from the Depth camera [ http://channel9.msdn.com/Series/KinectSDKQuickstarts/Camera-Fundamentals#time=13m06s 13:06 ] Adjusting camera tilt <h3>Setup</h3> The steps below assume you have setup your development environment as explained in the " http://channel9.msdn.com/Series/KinectSDKQuickstarts/Getting-Started Setting Up Your Development Environment " video. To get a depth or color image you must do the following <ol>Create an instance of a runtime object Initialize the instance with the correct options Subscribe to an event to receive camera data Open the video stream Convert the raw camera data into an image </ol><h1>Task: Display the RGB Camera Image</h1> Designing your UI We’ll add two 320x240 Image controls to the MainWindow XAML file as shown in the following XAML: XAML
<pre class="brush: xml <Window x:Class="WpfApplication1.MainWindow"
xmlns=" http://schemas.microsoft.com/winfx/2006/xaml/presentation" http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x=" http://schemas.microsoft.com/winfx/2006/xaml" http://schemas.microsoft.com/winfx/2006/xaml"
Title="MainWindow" Height="350" Width="800
<Grid>
<Image Height="240" HorizontalAlignment="Left" Margin="12,20,0,0" Name="image1" Stretch="Fill" VerticalAlignment="Top" Width="320" />
<Image Height="240" HorizontalAlignment="Left" Margin="355,20,0,0" Name="image2" Stretch="Fill" VerticalAlignment="Top" Width="320" />
</Grid>
</Window>[/code] <img title="image" src="http://files.channel9.msdn.com/wlwimages/9c00b398b405423b99d19efa016fae96/image%5B2%5D.png" alt="image" width="640" height="290" border="0 <h3>Initializing the runtime</h3> In the Window_Loaded event, initialize the runtime with the options you want to use. For this example, set RuntimeOptions.UseColor to use the RGB camera: C#
<pre class="brush: csharp nui.Initialize(RuntimeOptions.UseColor);[/code] Visual Basic
<pre class="brush: vb nui.Initialize(RuntimeOptions.UseColor)[/code] <h3>Subscribe to the VideoFrameReady event</h3> To read values from the RGB camera, add an event handler for the VideoFrameReady event on the Kinect Runtime in the Window_Loaded event as shown below: <blockquote> Tip: You can have Visual Studio automatically build the event handler for you by typing " nui.VideoFrameReady +=" and hitting the tab key twice. </blockquote> C#
<pre class="brush: csharp nui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);
void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
// event code goes here
}[/code] Visual Basic
<pre class="brush: vb AddHandler nui.VideoFrameReady, AddressOf nui_VideoFrameReady
Private Sub nui_VideoFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs)
event code goes here
End Sub[/code] Understanding the video frame ready returned values The video frame returns an ImageFrameReadyEventArgs that contains an ImageFrame class. As shown below, the ImageFrame class contains two things: Metadata about the image, such as ImageType to know if it’s a depth or color image, and resolution to know the size of the image Image – the actual image data, which is stored in the Bits byte[] array http://files.channel9.msdn.com/wlwimages/9c00b398b405423b99d19efa016fae96/image%5B3%5D.png <img title="image" src="http://files.channel9.msdn.com/wlwimages/9c00b398b405423b99d19efa016fae96/image_thumb%5B1%5D.png" alt="image" width="640" height="220" border="0 Converting the byte[] array to an image To convert the byte[] array that represents the camera image to display it in an Image control (ex: image1 below), call the BitmapSource.Create method as shown below. The last parameter is stride. The stride is the number of bytes from one row of pixels in memory to the next row of pixels in memory. Stride is also called pitch. For more information, go to <a title="http://msdn.microsoft.com/en-us/library/aa473780(VS.85).aspx http://msdn.microsoft.com/en-us/library/aa473780(VS.85).aspx http://msdn.microsoft.com/en-us/library/aa473780(VS.85).aspx : C#
<pre class="brush: csharp PlanarImage imageData = e.ImageFrame.Image;image1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96,
PixelFormats.Bgr32, null, imageData.Bits, data.Width * imageData.BytesPerPixel);[/code] Visual Basic
<pre class="brush: vb Dim imageData As PlanarImage = e.ImageFrame.Imageimage1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, _
PixelFormats.Bgr32, Nothing, data.Bits, imageData.Width * imageData.BytesPerPixel)[/code] The http://c4fkinect.codeplex.com/ Coding4Fun Kinect Toolkit has an extension method built into the ImageFrame class that simplifies creating the bitmap: C#
<pre class="brush: csharp image1.Source = e.ImageFrame.ToBitmapSource();[/code] Visual Basic
<pre class="brush: vb image1.Source = e.ImageFrame.ToBitmapSource()[/code] <h3>Getting Supported Resolutions</h3> The Kinect cameras can support different resolutions depending on what ImageType is passed in. To determine these resolutions, call the GetValidResolutions on the ImageStream object: C#
<pre class="brush: csharp var x = ImageStream.GetValidResolutions(ImageType.DepthAndPlayerIndex);[/code] Visual Basic
<pre class="brush: vb Dim x = ImageStream.GetValidResolutions(ImageType.DepthAndPlayerIndex)[/code] Open the VideoStream Before the event will fire, you must call the Open method on VideoStream as shown below. This opens a video stream, sets the PoolSize to 2, sets the resolution to 640x480, and sets the return image as a color image: C#
<pre class="brush: csharp nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);[/code] Visual Basic
<pre class="brush: vb nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color)[/code] Understanding PoolSize Pool Size is the number of buffers you want queued up. Usually, you want one displaying, and one being filled. The minimum is two but specifying more than two will give you slightly more latency in the frames you display, and so you’ll be more likely to have one waiting for you. The trade off, then, is smoother playback but higher latency. <h1>Task: Displaying Depth Data</h1> To add depth data, we use the same steps as above. <h3>Update the Runtime Initialization</h3> First, we’ll update the Initialize method to include RuntimeOptions.UseDepth to receive depth data from Kinect: C# <pre class="brush: csharp nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);[/code] Visual Basic
<pre class="brush: vb nui.Initialize(RuntimeOptions.UseColor Or RuntimeOptions.UseDepth)[/code] <h3>Subscribe to the DepthFrameReady event</h3> Next, add the event handler to receive the Depth information. For simplicity, we’ll use the Coding4Fun Kinect toolkit to display the depth image in the image2 Image control: C# <pre class="brush: csharp nui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);
void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
{
image2.Source = e.ImageFrame.ToBitmapSource();
}[/code] Visual Basic
<pre class="brush: vb AddHandler nui.DepthFrameReady, AddressOf nui_DepthFrameReady
Private Sub nui_DepthFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs)
image2.Source = e.ImageFrame.ToBitmapSource()
End Sub[/code] <h3>Open the Depth Stream</h3> Finally, we’ll call DepthStream.Open to open a 320x240 depth stream: C# <pre class="brush: csharp nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth);[/code] Visual Basic
<pre class="brush: vb nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth)[/code] The result is shown below: http://files.channel9.msdn.com/thumbnail/f6579560-3ab8-4997-a4d4-c150801b8e03.PNG" rel="lightbox <img src="http://files.channel9.msdn.com/thumbnail/f6579560-3ab8-4997-a4d4-c150801b8e03.PNG" alt=" <h1>Task: Adjust the Kinect Camera Tilt</h1><blockquote> Warning: The tilt mechanism in the sensor array is not rated for frequent use. Your code should not make calls to tilt the device more than 15 times in any two-minute window. Changing the tilt too often results in an error message from the function. </blockquote><h3>Setup</h3> Like before, create an instance of the Kinect Runtime class and call the Initialize method: C# <pre class="brush: csharp Runtime nui = new Runtime();nui.Initialize(RuntimeOptions.UseColor);[/code] Visual Basic
<pre class="brush: vb nui.Initialize(RuntimeOptions.UseColor)[/code] <h3>Adjusting the tilt</h3> To adjust the tilt or pitch, set the ElevationAngle property to a value between –27 ( Camera.ElevationMinimum ) and +27 ( Camera.ElevationMaximum ). The code below sets the camera to the maximum elevation: C# <pre class="brush: csharp nui.NuiCamera.ElevationAngle = Camera.ElevationMaximum;[/code] Visual Basic
<pre class="brush: vb nui.NuiCamera.ElevationAngle = Camera.ElevationMaximum[/code] <img src="http://m.webtrends.com/dcs1wotjh10000w0irc493s0e_6x1g/njs.gif?dcssip=channel9.msdn.com&dcsuri=http://channel9.msdn.com/Feeds/RSS&WT.dl=0&WT.entryid=Entry:RSSView:4ef787820c27406f89d89efc0065168e
View the full article
<pre class="brush: xml <Window x:Class="WpfApplication1.MainWindow"
xmlns=" http://schemas.microsoft.com/winfx/2006/xaml/presentation" http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x=" http://schemas.microsoft.com/winfx/2006/xaml" http://schemas.microsoft.com/winfx/2006/xaml"
Title="MainWindow" Height="350" Width="800
<Grid>
<Image Height="240" HorizontalAlignment="Left" Margin="12,20,0,0" Name="image1" Stretch="Fill" VerticalAlignment="Top" Width="320" />
<Image Height="240" HorizontalAlignment="Left" Margin="355,20,0,0" Name="image2" Stretch="Fill" VerticalAlignment="Top" Width="320" />
</Grid>
</Window>[/code] <img title="image" src="http://files.channel9.msdn.com/wlwimages/9c00b398b405423b99d19efa016fae96/image%5B2%5D.png" alt="image" width="640" height="290" border="0 <h3>Initializing the runtime</h3> In the Window_Loaded event, initialize the runtime with the options you want to use. For this example, set RuntimeOptions.UseColor to use the RGB camera: C#
<pre class="brush: csharp nui.Initialize(RuntimeOptions.UseColor);[/code] Visual Basic
<pre class="brush: vb nui.Initialize(RuntimeOptions.UseColor)[/code] <h3>Subscribe to the VideoFrameReady event</h3> To read values from the RGB camera, add an event handler for the VideoFrameReady event on the Kinect Runtime in the Window_Loaded event as shown below: <blockquote> Tip: You can have Visual Studio automatically build the event handler for you by typing " nui.VideoFrameReady +=" and hitting the tab key twice. </blockquote> C#
<pre class="brush: csharp nui.VideoFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_VideoFrameReady);
void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
// event code goes here
}[/code] Visual Basic
<pre class="brush: vb AddHandler nui.VideoFrameReady, AddressOf nui_VideoFrameReady
Private Sub nui_VideoFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs)
event code goes here
End Sub[/code] Understanding the video frame ready returned values The video frame returns an ImageFrameReadyEventArgs that contains an ImageFrame class. As shown below, the ImageFrame class contains two things: Metadata about the image, such as ImageType to know if it’s a depth or color image, and resolution to know the size of the image Image – the actual image data, which is stored in the Bits byte[] array http://files.channel9.msdn.com/wlwimages/9c00b398b405423b99d19efa016fae96/image%5B3%5D.png <img title="image" src="http://files.channel9.msdn.com/wlwimages/9c00b398b405423b99d19efa016fae96/image_thumb%5B1%5D.png" alt="image" width="640" height="220" border="0 Converting the byte[] array to an image To convert the byte[] array that represents the camera image to display it in an Image control (ex: image1 below), call the BitmapSource.Create method as shown below. The last parameter is stride. The stride is the number of bytes from one row of pixels in memory to the next row of pixels in memory. Stride is also called pitch. For more information, go to <a title="http://msdn.microsoft.com/en-us/library/aa473780(VS.85).aspx http://msdn.microsoft.com/en-us/library/aa473780(VS.85).aspx http://msdn.microsoft.com/en-us/library/aa473780(VS.85).aspx : C#
<pre class="brush: csharp PlanarImage imageData = e.ImageFrame.Image;image1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96,
PixelFormats.Bgr32, null, imageData.Bits, data.Width * imageData.BytesPerPixel);[/code] Visual Basic
<pre class="brush: vb Dim imageData As PlanarImage = e.ImageFrame.Imageimage1.Source = BitmapSource.Create(imageData.Width, imageData.Height, 96, 96, _
PixelFormats.Bgr32, Nothing, data.Bits, imageData.Width * imageData.BytesPerPixel)[/code] The http://c4fkinect.codeplex.com/ Coding4Fun Kinect Toolkit has an extension method built into the ImageFrame class that simplifies creating the bitmap: C#
<pre class="brush: csharp image1.Source = e.ImageFrame.ToBitmapSource();[/code] Visual Basic
<pre class="brush: vb image1.Source = e.ImageFrame.ToBitmapSource()[/code] <h3>Getting Supported Resolutions</h3> The Kinect cameras can support different resolutions depending on what ImageType is passed in. To determine these resolutions, call the GetValidResolutions on the ImageStream object: C#
<pre class="brush: csharp var x = ImageStream.GetValidResolutions(ImageType.DepthAndPlayerIndex);[/code] Visual Basic
<pre class="brush: vb Dim x = ImageStream.GetValidResolutions(ImageType.DepthAndPlayerIndex)[/code] Open the VideoStream Before the event will fire, you must call the Open method on VideoStream as shown below. This opens a video stream, sets the PoolSize to 2, sets the resolution to 640x480, and sets the return image as a color image: C#
<pre class="brush: csharp nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);[/code] Visual Basic
<pre class="brush: vb nui.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color)[/code] Understanding PoolSize Pool Size is the number of buffers you want queued up. Usually, you want one displaying, and one being filled. The minimum is two but specifying more than two will give you slightly more latency in the frames you display, and so you’ll be more likely to have one waiting for you. The trade off, then, is smoother playback but higher latency. <h1>Task: Displaying Depth Data</h1> To add depth data, we use the same steps as above. <h3>Update the Runtime Initialization</h3> First, we’ll update the Initialize method to include RuntimeOptions.UseDepth to receive depth data from Kinect: C# <pre class="brush: csharp nui.Initialize(RuntimeOptions.UseColor | RuntimeOptions.UseDepth);[/code] Visual Basic
<pre class="brush: vb nui.Initialize(RuntimeOptions.UseColor Or RuntimeOptions.UseDepth)[/code] <h3>Subscribe to the DepthFrameReady event</h3> Next, add the event handler to receive the Depth information. For simplicity, we’ll use the Coding4Fun Kinect toolkit to display the depth image in the image2 Image control: C# <pre class="brush: csharp nui.DepthFrameReady += new EventHandler<ImageFrameReadyEventArgs>(nui_DepthFrameReady);
void nui_DepthFrameReady(object sender, ImageFrameReadyEventArgs e)
{
image2.Source = e.ImageFrame.ToBitmapSource();
}[/code] Visual Basic
<pre class="brush: vb AddHandler nui.DepthFrameReady, AddressOf nui_DepthFrameReady
Private Sub nui_DepthFrameReady(ByVal sender As Object, ByVal e As ImageFrameReadyEventArgs)
image2.Source = e.ImageFrame.ToBitmapSource()
End Sub[/code] <h3>Open the Depth Stream</h3> Finally, we’ll call DepthStream.Open to open a 320x240 depth stream: C# <pre class="brush: csharp nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth);[/code] Visual Basic
<pre class="brush: vb nui.DepthStream.Open(ImageStreamType.Depth, 2, ImageResolution.Resolution320x240, ImageType.Depth)[/code] The result is shown below: http://files.channel9.msdn.com/thumbnail/f6579560-3ab8-4997-a4d4-c150801b8e03.PNG" rel="lightbox <img src="http://files.channel9.msdn.com/thumbnail/f6579560-3ab8-4997-a4d4-c150801b8e03.PNG" alt=" <h1>Task: Adjust the Kinect Camera Tilt</h1><blockquote> Warning: The tilt mechanism in the sensor array is not rated for frequent use. Your code should not make calls to tilt the device more than 15 times in any two-minute window. Changing the tilt too often results in an error message from the function. </blockquote><h3>Setup</h3> Like before, create an instance of the Kinect Runtime class and call the Initialize method: C# <pre class="brush: csharp Runtime nui = new Runtime();nui.Initialize(RuntimeOptions.UseColor);[/code] Visual Basic
<pre class="brush: vb nui.Initialize(RuntimeOptions.UseColor)[/code] <h3>Adjusting the tilt</h3> To adjust the tilt or pitch, set the ElevationAngle property to a value between –27 ( Camera.ElevationMinimum ) and +27 ( Camera.ElevationMaximum ). The code below sets the camera to the maximum elevation: C# <pre class="brush: csharp nui.NuiCamera.ElevationAngle = Camera.ElevationMaximum;[/code] Visual Basic
<pre class="brush: vb nui.NuiCamera.ElevationAngle = Camera.ElevationMaximum[/code] <img src="http://m.webtrends.com/dcs1wotjh10000w0irc493s0e_6x1g/njs.gif?dcssip=channel9.msdn.com&dcsuri=http://channel9.msdn.com/Feeds/RSS&WT.dl=0&WT.entryid=Entry:RSSView:4ef787820c27406f89d89efc0065168e
View the full article