Reading 4GB file

[dudeSreeni]

Hi,

I need to read a 4GB log file to extract some information and write it to another file.

Can any one suggest me an ideal class i should use to read the file using C#.


Im planning to use Regular Expressions to match the lines that contain the parameter strings.

currently the file is read and written using commands in DOS and it takes around 15 - 18 mins.

Im planning to make it much quicker with the application im planning to develop.

My development machine has 2 GB of RAM.

Can any one advise me on this.

Thanks and Regards,

Sree

 

[PlausiblyDamp]

Could you not simply use a StreamReader and read the file a line at a time and check each line as you read it?

If performance suffers a lnie at a time then you might find a FileStream will allow you to read chunks into memory and then use a stream reader over the memory you have just read in.

Another possibility is using memory mapped files, however this isnt as straight forward from .Net although google will find several hits that will point you in the right direction.

 

[Nate Bross]

As PD suggests, you might try something like this:

StreamReader reader = new StreamReader(@"C:\your4GBFile.log",...more parms);
while ((String line = reader.ReadLine()) != null) 
    System.WriteLine(line);
reader.Close();

 

[Nate Bross]

Another thing worth keeping track of is string concatenation. While you are building up your second string in memory, that you will write out at the end, be sure to use the StringBuilder class.

...code...
StringBuilder sb = new StringBuilder();
sb.Append(line);
...code...

This code above is MUCH faster than this

...code...
String s = "";
s += line;
...code...