EDN Admin
Well-known member
Good morning everyone:
This days Ive been trying to build a proof of concept program that uploads big files into a web service. I took many roads to make it to the one I though was the right one. Im currently using webClient to write a SOAP message to the web service (because
of many exceptions due to the high memory consumption) and the real problem now seems that the stream returned from the WebClient is not flushing, but retaining the whole data in memory before sending it, and I dont know why.
After profiling my application memory usage, I found that the Webclient returned stream is all buffered in memory, and calling "Flush()" does not make a thing, so the stream is taking large amounts of memory and the problem are not my chunks or temporary
buffers (which I blamed for hours) but the internal stream....
So my question is, Is there anyway to make this stream flush and not take the whole memory???? Or this is a lost cause and I better move to an other approach (by the time I have no idea where to go).
A code snippet in here of the upload method:
<div style="color:Black;background-color:White; <pre>
WebClient client = <span style="color:Blue; new WebClient();
Stream postStream = client.OpenWrite(uriString);
FileStream fs = File.OpenRead(<span style="color:#A31515; @"C:TempSharepoint2003-2.zip");
<span style="color:Blue; int chunkSize = 3000;
<span style="color:Blue; byte[] buffer = <span style="color:Blue; new <span style="color:Blue; byte[chunkSize];
<span style="color:Blue; int readBytes = fs.Read(buffer, 0, buffer.Length);
<span style="color:Blue; long fsBytes = fs.Length;
<span style="color:Blue; int cycles = 0;
<span style="color:Blue; byte[] encodedByteArray;
<span style="color:Blue; string chars;
<span style="color:Blue; while (readBytes > 0)
{
chars = Convert.ToBase64String(buffer, 0, readBytes);
encodedByteArray = System.Text.Encoding.ASCII.GetBytes( chars);
chars = <span style="color:Blue; null;
Array.Clear(buffer, 0, buffer.Length);
postStream.Write(encodedByteArray , 0, encodedByteArray.Length);
postStream.Flush(); <span style="color:Green; //IN HERE: all the memory allocated, not releasing anything... the stream is just getting bigger
Array.Clear(encodedByteArray, 0, encodedByteArray.Length);
readBytes = fs.Read(buffer, 0, buffer.Length);
<span style="color:Blue; if (cycles == 1000) {
GC.GetTotalMemory(<span style="color:Blue; true);
}
cycles++;
}
fs.Close();
[/code]
Any help will be appreciated... <hr class="sig I need to do, I need to know.
View the full article
This days Ive been trying to build a proof of concept program that uploads big files into a web service. I took many roads to make it to the one I though was the right one. Im currently using webClient to write a SOAP message to the web service (because
of many exceptions due to the high memory consumption) and the real problem now seems that the stream returned from the WebClient is not flushing, but retaining the whole data in memory before sending it, and I dont know why.
After profiling my application memory usage, I found that the Webclient returned stream is all buffered in memory, and calling "Flush()" does not make a thing, so the stream is taking large amounts of memory and the problem are not my chunks or temporary
buffers (which I blamed for hours) but the internal stream....
So my question is, Is there anyway to make this stream flush and not take the whole memory???? Or this is a lost cause and I better move to an other approach (by the time I have no idea where to go).
A code snippet in here of the upload method:
<div style="color:Black;background-color:White; <pre>
WebClient client = <span style="color:Blue; new WebClient();
Stream postStream = client.OpenWrite(uriString);
FileStream fs = File.OpenRead(<span style="color:#A31515; @"C:TempSharepoint2003-2.zip");
<span style="color:Blue; int chunkSize = 3000;
<span style="color:Blue; byte[] buffer = <span style="color:Blue; new <span style="color:Blue; byte[chunkSize];
<span style="color:Blue; int readBytes = fs.Read(buffer, 0, buffer.Length);
<span style="color:Blue; long fsBytes = fs.Length;
<span style="color:Blue; int cycles = 0;
<span style="color:Blue; byte[] encodedByteArray;
<span style="color:Blue; string chars;
<span style="color:Blue; while (readBytes > 0)
{
chars = Convert.ToBase64String(buffer, 0, readBytes);
encodedByteArray = System.Text.Encoding.ASCII.GetBytes( chars);
chars = <span style="color:Blue; null;
Array.Clear(buffer, 0, buffer.Length);
postStream.Write(encodedByteArray , 0, encodedByteArray.Length);
postStream.Flush(); <span style="color:Green; //IN HERE: all the memory allocated, not releasing anything... the stream is just getting bigger
Array.Clear(encodedByteArray, 0, encodedByteArray.Length);
readBytes = fs.Read(buffer, 0, buffer.Length);
<span style="color:Blue; if (cycles == 1000) {
GC.GetTotalMemory(<span style="color:Blue; true);
}
cycles++;
}
fs.Close();
[/code]
Any help will be appreciated... <hr class="sig I need to do, I need to know.
View the full article