EDN Admin
Well-known member
Last year the Coding4Fun/Channel 9 guys asked me to work on a few things for http://live.visitmix.com/" target="_blank MIX10 . One of these items was a way to output a webcam stream to Windows Phone 7 for use with Clints http://blogs.msdn.com/b/coding4fun/archive/2010/03/16/9979874.aspx" target="_blank t-shirt cannon project you may have read about. I figured the easiest way to accomplish this was by using a network/IP camera capable of sending a Motion-JPEG stream, which can be easily decoded and displayed that can display a JPEG image. Thus, this library was born. It has gone through quite a few changes and I have expanded it to easily display MJPEG streams on a variety of platforms. The developer just references the assembly appropriate to their platform, adds a few lines of code, and away it goes. Usage For those that are just interested in the usage, its as simple as this: <ol>Reference one of the following assemblies appropriate for your project: MjpegProcessorSL.dll - Silverlight ( Out of Browser Only! ) MjpegProcessorWP7.dll - Windows Phone 7 (XNA or Silverlight, performance maxes out around 320x240 @ 15fps, so set your camera settings accordingly) MjpegProcessorXna4.dll - XNA 4.0 (Windows) MjpegProcessor.dll - WinForms and WPF Create a new MjpegDecoder object. Hook up the FrameReady event. In the event handler, take the Bitmap / BitmapImage and assign it to your image display control: In the case of XNA, use the GetMjpegFrame method in the Update method, which will return a Texture2D you can use in your Draw method. Call the ParseStream method with the Uri of the MJPEG "endpoint". </ol> Thats it! The source code and binaries above both include projects demonstrating how to use the library on each of these platforms. As long as you set the appropriate reference, you can just copy and paste the code in the sample to get your project running (changing the Uri, of course). <pre class="brush: csharp; public partial class MainWindow : Window
{
MjpegDecoder _mjpeg;
public MainWindow()
{
InitializeComponent();
_mjpeg = new MjpegDecoder();
_mjpeg.FrameReady += mjpeg_FrameReady;
}
private void Start_Click(object sender, RoutedEventArgs e)
{
_mjpeg.ParseStream(new Uri("http://192.168.2.200/img/video.mjpeg"));
}
private void mjpeg_FrameReady(object sender, FrameReadyEventArgs e)
{
image.Source = e.BitmapImage;
}
}[/code] If that doesnt fit your needs, you can also access the Bitmap/BitmapImage properties directly from the MjpegDecoder object, or the CurrentFrame property, which will contain the raw JPEG data prior to being decoded. A Word About Network/IP Cameras I have tested this against several different cameras. Each device has its own quirks, but all of them seem to work with this library with one exception: several cameras will respond differently when an Internet Explorer user agent header is sent with the HTTP request. Instead of sending down an MJPEG stream, it will send a single JPEG image as Internet Explorer does not properly support MJPEG streams. Unfortunately, this causes the Silverlight processor to not work properly as the header cannot be changed from the Internet Explorer default. When this happens, only a single frame will be sent, and the decoding will fail. The only fix I have found is to use a different camera that doesnt work in this way. What is MJPEG? Pretty simply, its a video format where each frame of video is sent as a separate, compressed JPEG image. A standard HTTP request is made to a specific URL, and a multipart response is sent. Parsing this multipart stream into separate images as they are sent results in a series of JPEG images. The viewer displays those JPEG images as quickly as they are sent and that creates the video. Its not a well documented format, nor is it perfectly standardized, but it does work. For more information, see the http://en.wikipedia.org/wiki/Motion_JPEG" target="_blank MJPEG article on Wikipedia . How Do I Find the MJPEG URL of My Camera? Excellent question. Not an excellent answer. The user manual may mention the URL. A quick internet search with the model number should get you a result. Or, you can also try this companys http://skjm.com/icam/mjpeg.php" target="_blank lookup tool . How Does It Work? Glad you asked. If you take a look at the project, youll notice there isnt much code. One single file is used with a variety of compiler directives to compile certain portions based on the platform assembly being generated. The MjpegDecoder.cs/.vb contains the entire implementation. First, an asynchronous request is made to the provided MJPEG URL inside the ParseStream method. If we are in a Silverlight environment, the AllowReadStreamBuffering property must be set to false so the response returns immediately instead of being buffered. Additionally, we need to register the http:// prefix to use the client http stack vs. the browser stack. Finally, the request is made using the BeginGetResponse method, specifying the OnGetResponse method as the callback. This will be called as soon as data is sent from the camera in response to our request. <pre class="brush: csharp; public void ParseStream(Uri uri)
{
#if SILVERLIGHT
HttpWebRequest.RegisterPrefix("http://", WebRequestCreator.ClientHttp);
#endif
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);
#if SILVERLIGHT
// start the stream immediately
request.AllowReadStreamBuffering = false;
#endif
// asynchronously get a response
request.BeginGetResponse(OnGetResponse, request);
}[/code] OnGetResponse grabs the response headers and uses the Content-Type header to determine the boundary marker that will be sent between JPEG frames. <pre class="brush: csharp; private void OnGetResponse(IAsyncResult asyncResult)
{
HttpWebResponse resp;
byte[] buff;
byte[] imageBuffer = new byte[1024 * 1024];
Stream s;
// get the response
HttpWebRequest req = (HttpWebRequest)asyncResult.AsyncState;
resp = (HttpWebResponse)req.EndGetResponse(asyncResult);
// find our magic boundary value
string contentType = resp.Headers["Content-Type"];
if(!string.IsNullOrEmpty(contentType) && !contentType.Contains("="))
throw new Exception("Invalid content-type header. The camera is likely not returning a proper MJPEG stream.");
string boundary = resp.Headers["Content-Type"].Split(=)[1];
byte[] boundaryBytes = Encoding.UTF8.GetBytes(boundary.StartsWith("--") ? boundary : "--" + boundary);
...[/code] It then streams the response data, looks for a JPEG header marker, then reads until it finds the boundary marker, copies the data into a buffer, decodes it, passes it on to whoever wants it via an event, and then starts over. <pre class="brush: csharp; ...
s = resp.GetResponseStream();
BinaryReader br = new BinaryReader(s);
_streamActive = true;
buff = br.ReadBytes(ChunkSize);
while (_streamActive)
{
int size;
// find the JPEG header
int imageStart = buff.Find(JpegHeader);
if(imageStart != -1)
{
// copy the start of the JPEG image to the imageBuffer
size = buff.Length - imageStart;
Array.Copy(buff, imageStart, imageBuffer, 0, size);
while(true)
{
buff = br.ReadBytes(ChunkSize);
// find the boundary text
int imageEnd = buff.Find(boundaryBytes);
if(imageEnd != -1)
{
// copy the remainder of the JPEG to the imageBuffer
Array.Copy(buff, 0, imageBuffer, size, imageEnd);
size += imageEnd;
// create a single JPEG frame
CurrentFrame = new byte[size];
Array.Copy(imageBuffer, 0, CurrentFrame, 0, size);
#if !XNA
ProcessFrame(CurrentFrame);
#endif
// copy the leftover data to the start
Array.Copy(buff, imageEnd, buff, 0, buff.Length - imageEnd);
// fill the remainder of the buffer with new data and start over
byte[] temp = br.ReadBytes(imageEnd);
Array.Copy(temp, 0, buff, buff.Length - imageEnd, temp.Length);
break;
}
// copy all of the data to the imageBuffer
Array.Copy(buff, 0, imageBuffer, size, buff.Length);
size += buff.Length;
}
}
}
resp.Close();
}[/code] The ProcessFrame method seen above takes the raw byte buffer, which contains an undecoded JPEG image, and decodes it based on the environment. However this isnt called in the case of XNA which well see in a moment: <pre class="brush: csharp; private void ProcessFrame(byte[] frameBuffer)
{
#if SILVERLIGHT
// need to get this back on the UI thread
Deployment.Current.Dispatcher.BeginInvoke((Action)(() =>
{
// resets the BitmapImage to the new frame
BitmapImage.SetSource(new MemoryStream(frameBuffer, 0, frameBuffer.Length));
// tell whoevers listening that we have a frame to draw
if(FrameReady != null)
FrameReady(this, new FrameReadyEventArgs { FrameBuffer = CurrentFrame, BitmapImage = BitmapImage });
}));
#endif
#if !SILVERLIGHT && !XNA
// I assume if theres an Application.Current then were in WPF, not WinForms
if(Application.Current != null)
{
// get it on the UI thread
Application.Current.Dispatcher.BeginInvoke((Action)(() =>
{
// create a new BitmapImage from the JPEG bytes
BitmapImage = new BitmapImage();
BitmapImage.BeginInit();
BitmapImage.StreamSource = new MemoryStream(frameBuffer);
BitmapImage.EndInit();
// tell whoevers listening that we have a frame to draw
if(FrameReady != null)
FrameReady(this, new FrameReadyEventArgs { FrameBuffer = CurrentFrame, Bitmap = Bitmap, BitmapImage = BitmapImage });
}));
}
else
{
// create a simple GDI+ happy Bitmap
Bitmap = new Bitmap(new MemoryStream(frameBuffer));
// tell whoevers listening that we have a frame to draw
if(FrameReady != null)
FrameReady(this, new FrameReadyEventArgs { FrameBuffer = CurrentFrame, Bitmap = Bitmap, BitmapImage = BitmapImage });
}
#endif
}[/code] In the case of Silverlight, the BitmapImage object has a SetSource method which takes a stream to be decoded and turned into the image. In WPF, BitmapImage works differently. In this case, BeginInit is called, then the StreamSource property is set to the stream of bytes, and finally EndInit is called. In WinForms, the library will return a Bitmap object which can be initialized with the stream right in the constructor. In the code above, I look at the Application.Current property to determine if the library is being used by a WPF project. If that property is not null, it is assumed the library is being called from a WPF project. When compiled as an XNA library, we have no use for a BitmapImage or a Bitmap …we need a Texture2D object. The GetMjpegFrame method seen below is what is called by an XNA application during the Update method to pull the current frame: <pre class="brush: csharp; public Texture2D GetMjpegFrame(GraphicsDevice graphicsDevice)
{
// create a Texture2D from the current byte buffer
if(CurrentFrame != null)
return Texture2D.FromStream(graphicsDevice, new MemoryStream(CurrentFrame, 0, CurrentFrame.Length));
return null;
}[/code] Conclusion And thats about it for the library. Give it a try and please contact me with feedback or if you run into any issues. Enjoy! <img src="http://m.webtrends.com/dcs1wotjh10000w0irc493s0e_6x1g/njs.gif?dcssip=channel9.msdn.com&dcsuri=http://channel9.msdn.com/Feeds/RSS&WT.dl=0&WT.entryid=Entry:RSSView:c0c99db087794de090759e86005409c8
View the full article
{
MjpegDecoder _mjpeg;
public MainWindow()
{
InitializeComponent();
_mjpeg = new MjpegDecoder();
_mjpeg.FrameReady += mjpeg_FrameReady;
}
private void Start_Click(object sender, RoutedEventArgs e)
{
_mjpeg.ParseStream(new Uri("http://192.168.2.200/img/video.mjpeg"));
}
private void mjpeg_FrameReady(object sender, FrameReadyEventArgs e)
{
image.Source = e.BitmapImage;
}
}[/code] If that doesnt fit your needs, you can also access the Bitmap/BitmapImage properties directly from the MjpegDecoder object, or the CurrentFrame property, which will contain the raw JPEG data prior to being decoded. A Word About Network/IP Cameras I have tested this against several different cameras. Each device has its own quirks, but all of them seem to work with this library with one exception: several cameras will respond differently when an Internet Explorer user agent header is sent with the HTTP request. Instead of sending down an MJPEG stream, it will send a single JPEG image as Internet Explorer does not properly support MJPEG streams. Unfortunately, this causes the Silverlight processor to not work properly as the header cannot be changed from the Internet Explorer default. When this happens, only a single frame will be sent, and the decoding will fail. The only fix I have found is to use a different camera that doesnt work in this way. What is MJPEG? Pretty simply, its a video format where each frame of video is sent as a separate, compressed JPEG image. A standard HTTP request is made to a specific URL, and a multipart response is sent. Parsing this multipart stream into separate images as they are sent results in a series of JPEG images. The viewer displays those JPEG images as quickly as they are sent and that creates the video. Its not a well documented format, nor is it perfectly standardized, but it does work. For more information, see the http://en.wikipedia.org/wiki/Motion_JPEG" target="_blank MJPEG article on Wikipedia . How Do I Find the MJPEG URL of My Camera? Excellent question. Not an excellent answer. The user manual may mention the URL. A quick internet search with the model number should get you a result. Or, you can also try this companys http://skjm.com/icam/mjpeg.php" target="_blank lookup tool . How Does It Work? Glad you asked. If you take a look at the project, youll notice there isnt much code. One single file is used with a variety of compiler directives to compile certain portions based on the platform assembly being generated. The MjpegDecoder.cs/.vb contains the entire implementation. First, an asynchronous request is made to the provided MJPEG URL inside the ParseStream method. If we are in a Silverlight environment, the AllowReadStreamBuffering property must be set to false so the response returns immediately instead of being buffered. Additionally, we need to register the http:// prefix to use the client http stack vs. the browser stack. Finally, the request is made using the BeginGetResponse method, specifying the OnGetResponse method as the callback. This will be called as soon as data is sent from the camera in response to our request. <pre class="brush: csharp; public void ParseStream(Uri uri)
{
#if SILVERLIGHT
HttpWebRequest.RegisterPrefix("http://", WebRequestCreator.ClientHttp);
#endif
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(uri);
#if SILVERLIGHT
// start the stream immediately
request.AllowReadStreamBuffering = false;
#endif
// asynchronously get a response
request.BeginGetResponse(OnGetResponse, request);
}[/code] OnGetResponse grabs the response headers and uses the Content-Type header to determine the boundary marker that will be sent between JPEG frames. <pre class="brush: csharp; private void OnGetResponse(IAsyncResult asyncResult)
{
HttpWebResponse resp;
byte[] buff;
byte[] imageBuffer = new byte[1024 * 1024];
Stream s;
// get the response
HttpWebRequest req = (HttpWebRequest)asyncResult.AsyncState;
resp = (HttpWebResponse)req.EndGetResponse(asyncResult);
// find our magic boundary value
string contentType = resp.Headers["Content-Type"];
if(!string.IsNullOrEmpty(contentType) && !contentType.Contains("="))
throw new Exception("Invalid content-type header. The camera is likely not returning a proper MJPEG stream.");
string boundary = resp.Headers["Content-Type"].Split(=)[1];
byte[] boundaryBytes = Encoding.UTF8.GetBytes(boundary.StartsWith("--") ? boundary : "--" + boundary);
...[/code] It then streams the response data, looks for a JPEG header marker, then reads until it finds the boundary marker, copies the data into a buffer, decodes it, passes it on to whoever wants it via an event, and then starts over. <pre class="brush: csharp; ...
s = resp.GetResponseStream();
BinaryReader br = new BinaryReader(s);
_streamActive = true;
buff = br.ReadBytes(ChunkSize);
while (_streamActive)
{
int size;
// find the JPEG header
int imageStart = buff.Find(JpegHeader);
if(imageStart != -1)
{
// copy the start of the JPEG image to the imageBuffer
size = buff.Length - imageStart;
Array.Copy(buff, imageStart, imageBuffer, 0, size);
while(true)
{
buff = br.ReadBytes(ChunkSize);
// find the boundary text
int imageEnd = buff.Find(boundaryBytes);
if(imageEnd != -1)
{
// copy the remainder of the JPEG to the imageBuffer
Array.Copy(buff, 0, imageBuffer, size, imageEnd);
size += imageEnd;
// create a single JPEG frame
CurrentFrame = new byte[size];
Array.Copy(imageBuffer, 0, CurrentFrame, 0, size);
#if !XNA
ProcessFrame(CurrentFrame);
#endif
// copy the leftover data to the start
Array.Copy(buff, imageEnd, buff, 0, buff.Length - imageEnd);
// fill the remainder of the buffer with new data and start over
byte[] temp = br.ReadBytes(imageEnd);
Array.Copy(temp, 0, buff, buff.Length - imageEnd, temp.Length);
break;
}
// copy all of the data to the imageBuffer
Array.Copy(buff, 0, imageBuffer, size, buff.Length);
size += buff.Length;
}
}
}
resp.Close();
}[/code] The ProcessFrame method seen above takes the raw byte buffer, which contains an undecoded JPEG image, and decodes it based on the environment. However this isnt called in the case of XNA which well see in a moment: <pre class="brush: csharp; private void ProcessFrame(byte[] frameBuffer)
{
#if SILVERLIGHT
// need to get this back on the UI thread
Deployment.Current.Dispatcher.BeginInvoke((Action)(() =>
{
// resets the BitmapImage to the new frame
BitmapImage.SetSource(new MemoryStream(frameBuffer, 0, frameBuffer.Length));
// tell whoevers listening that we have a frame to draw
if(FrameReady != null)
FrameReady(this, new FrameReadyEventArgs { FrameBuffer = CurrentFrame, BitmapImage = BitmapImage });
}));
#endif
#if !SILVERLIGHT && !XNA
// I assume if theres an Application.Current then were in WPF, not WinForms
if(Application.Current != null)
{
// get it on the UI thread
Application.Current.Dispatcher.BeginInvoke((Action)(() =>
{
// create a new BitmapImage from the JPEG bytes
BitmapImage = new BitmapImage();
BitmapImage.BeginInit();
BitmapImage.StreamSource = new MemoryStream(frameBuffer);
BitmapImage.EndInit();
// tell whoevers listening that we have a frame to draw
if(FrameReady != null)
FrameReady(this, new FrameReadyEventArgs { FrameBuffer = CurrentFrame, Bitmap = Bitmap, BitmapImage = BitmapImage });
}));
}
else
{
// create a simple GDI+ happy Bitmap
Bitmap = new Bitmap(new MemoryStream(frameBuffer));
// tell whoevers listening that we have a frame to draw
if(FrameReady != null)
FrameReady(this, new FrameReadyEventArgs { FrameBuffer = CurrentFrame, Bitmap = Bitmap, BitmapImage = BitmapImage });
}
#endif
}[/code] In the case of Silverlight, the BitmapImage object has a SetSource method which takes a stream to be decoded and turned into the image. In WPF, BitmapImage works differently. In this case, BeginInit is called, then the StreamSource property is set to the stream of bytes, and finally EndInit is called. In WinForms, the library will return a Bitmap object which can be initialized with the stream right in the constructor. In the code above, I look at the Application.Current property to determine if the library is being used by a WPF project. If that property is not null, it is assumed the library is being called from a WPF project. When compiled as an XNA library, we have no use for a BitmapImage or a Bitmap …we need a Texture2D object. The GetMjpegFrame method seen below is what is called by an XNA application during the Update method to pull the current frame: <pre class="brush: csharp; public Texture2D GetMjpegFrame(GraphicsDevice graphicsDevice)
{
// create a Texture2D from the current byte buffer
if(CurrentFrame != null)
return Texture2D.FromStream(graphicsDevice, new MemoryStream(CurrentFrame, 0, CurrentFrame.Length));
return null;
}[/code] Conclusion And thats about it for the library. Give it a try and please contact me with feedback or if you run into any issues. Enjoy! <img src="http://m.webtrends.com/dcs1wotjh10000w0irc493s0e_6x1g/njs.gif?dcssip=channel9.msdn.com&dcsuri=http://channel9.msdn.com/Feeds/RSS&WT.dl=0&WT.entryid=Entry:RSSView:c0c99db087794de090759e86005409c8
View the full article