regexp using a ton of memory

[mattsrebatespam]

I am trying to search through a bunch of text files (~25K-50K files) using regular expressions. I have a FileContentsArray array that contains each line of the file. I also have a SearchExpression array that has usually < 10 regular expressions. The code is working, but the memory usage is enormous. The program ties up all the system resources and takes forever to finish. I have managed to isolate the huge memory usage to the second line where I declare the Regexp. Is there any way to not declare it as a new object or to purge the object each time? I have tried to set Regexp = nothing but the memory used remains the same. Any help is greatly appreciated.

For j = 0 To FileContentsArray.Count - 1

Dim Regexp As Regex = New Regex(SearchExpression.Item(k))
If Regexp.Match(FileContentsArray.Item(j)).Success Then

add the match to a dataset​

End If​

Next

 

[mattsrebatespam]

Another thing I would like to add is that if I comment out the If.. Then part of the code where the match is performed (leaving the declaration of the regex object only) the memory usage is still high.

I am trying to search through a bunch of text files (~25K-50K files) using regular expressions. I have a FileContentsArray array that contains each line of the file. I also have a SearchExpression array that has usually < 10 regular expressions. The code is working, but the memory usage is enormous. The program ties up all the system resources and takes forever to finish. I have managed to isolate the huge memory usage to the second line where I declare the Regexp. Is there any way to not declare it as a new object or to purge the object each time? I have tried to set Regexp = nothing but the memory used remains the same. Any help is greatly appreciated.

For j = 0 To FileContentsArray.Count - 1

Dim Regexp As Regex = New Regex(SearchExpression.Item(k))
If Regexp.Match(FileContentsArray.Item(j)).Success Then

add the match to a dataset​

End If​

Next

 

[seve7_wa]

Dont open all the files at once(!).

Im not at all familiar with VB so; heres an algorithm that should be slightly more efficient:

build an array, MyFilenamesArray, of valid file names
build an integer, sexpcount, equal to the last valid index value of your regular 
   expression array

count = index of MyFilenamesArray-1
for (iteration = 0..count) do
    InnerStreamReader = new sr (MyFilenamesArray.Item(iteration))
    InnerTextblock = read_to_end_of_file InnerStreamReader
    close InnerStreamReader
    for (rexp_iter = 0..sexpcount)
        Regexp = new Regex(SearchExpression.Item(rexp_iter))
        regex work here
        close/garbage collect regexp
    end_inner_for
    close/garbage collect innertextblock
end_for

 

[snarfblam]

Im not too familiar with regular expressions but looking at your code i noticed that you declare a new regex with every iteration. Thats a lot of iterations, so of course you will generate a lot of objects. If it is not necessary to create a new one each time then that can save you oodles of memory.

I also dont know a ton about the garbage collection, but Im not sure that it will occur in the middle of a function like that. I havent the slightest idea as to whether or not this will work but if you call doevents every now and then it might give the app a chance to do garbage collection. With an operation this big you want to call doevents every now and then just as to not hog 100% of the cpu. You could also call gc.collect(). Calls to gc.collect can cause memory to be managed worse later in the app though because you are forcing any objects still in use into later generations, so be careful with that one.

 

[Nerseus]

I see at least three things that can be made faster:
1. Use a foreach loop. A regular for loop incurrs extra overhead vs. a foreach as .NET will do an array bounds check on each iteration. This is small, but since a foreach is *easier*, might as well use it:

Dim s As String
ForEach  s in FileContentsArray
    Dim Regexp As Regex = New Regex(SearchExpression.Item(k))
    If Regexp.Match(s).Success Then
        add the match to a dataset
    End If
Next

2. Create the RegEx objects once, outside of the loop. Not sure what k is - Im guessing you loop through regular expressions? Below is a new version that just moves out RegEx. The GC will have to create the object on each iteration and then GC it. Thats a tremendous amount of overhead that you can avoid.

Dim s As String
Dim Regexp As Regex = New Regex(SearchExpression.Item(k))
ForEach  s in FileContentsArray
    If Regexp.Match(s).Success Then
        add the match to a dataset
    End If
Next

3. Put the match into a Match object so you can use it inside the If. This will help if youre testing for Success then getting the Match object into a variable to do something with the matches. You didnt show what you did with the data.

Dim s As String
Dim Regexp As Regex = New Regex(SearchExpression.Item(k))
Dim m As Match
ForEach  s in FileContentsArray
    m = Regexp.Match(s)
    If m.Success Then
        add the match to a dataset
        ds.Tables... = match.Match.Groups["Col1"].Value
    End If
Next

There could be lots of other optimaztions wed recommend. The best thing to do is profile the code yourself and determine EXACTLY what the slowest part of the code is. If you dont have a tool, some scattered Debug.Write statements should help. Its always better to analyze first before diving into code changes where you think its slow - its just too easy to find out for sure.

The first two above are pretty easy to do though, so you might try them first.

-ner

 

[mattsrebatespam]

Thanks for the advice. I knew about garbage collection but did not know how to use it. In my larger datasets I am doing 10 million+ searches (at about 10-15 thousand per second). It is just taking to long and I knew that there was a better way to manage the resources.
I will try out your suggestions and see if it helps, thanks again!