Pattern Finding in Files using Byte Arrays and Multiparted patterns to search for

[EDN Admin]

As a continuation of my other thread, "Finding A Pattern without a string Array (http://social.msdn.microsoft.com/Forums/en-US/vbgeneral/thread/5b3bd5f8-d1d3-4faa-a37e-4834cf721a31/)", I would like to continue asking questions about enhancing a scanner with
a few things:
1. Here is my code for finding signatures in a file:

<div style="color:Black;background-color:White; <pre>
<span style="color:Green; First the Signature Class:
<span style="color:Blue; Interface Signature
<span style="color:Blue; End <span style="color:Blue; Interface
<span style="color:Blue; Class Match : <span style="color:Blue; Implements Signature
<span style="color:Blue; End <span style="color:Blue; Class
<span style="color:Blue; Class binarySignature : <span style="color:Blue; Implements Signature
<span style="color:Blue; Public strfound <span style="color:Blue; As <span style="color:Blue; String
<span style="color:Blue; Public Ptrs <span style="color:Blue; As <span style="color:Blue; New Dictionary(Of <span style="color:Blue; Byte, Signature)
<span style="color:Blue; Public <span style="color:Blue; Shared <span style="color:Blue; Event MatchFound <span style="color:Blue; As EventHandler
<span style="color:Blue; Public <span style="color:Blue; Sub Search(<span style="color:Blue; ByRef arr <span style="color:Blue; As <span style="color:Blue; Byte(), <span style="color:Blue; ByVal I <span style="color:Blue; As <span style="color:Blue; Integer)
<span style="color:Blue; If <span style="color:Blue; Not I = arr.Length <span style="color:Blue; Then
<span style="color:Blue; If Ptrs.Keys.Contains(arr(I)) <span style="color:Blue; Then
<span style="color:Blue; If <span style="color:Blue; TypeOf Ptrs(arr(I)) <span style="color:Blue; Is Match <span style="color:Blue; Then
strfound = Ptrs(arr(I)).ToString
<span style="color:Blue; RaiseEvent MatchFound(<span style="color:Blue; Me, <span style="color:Blue; New EventArgs)
<span style="color:Blue; Else
<span style="color:Blue; Call <span style="color:Blue; DirectCast(Ptrs(arr(I)), binarySignature).Search(arr, I + 1)
<span style="color:Blue; End <span style="color:Blue; If
<span style="color:Blue; End <span style="color:Blue; If
<span style="color:Blue; End <span style="color:Blue; If
<span style="color:Blue; End <span style="color:Blue; Sub
<span style="color:Blue; WriteOnly <span style="color:Blue; Property AddSignaturePtr(<span style="color:Blue; ByVal <span style="color:Blue; Order <span style="color:Blue; As <span style="color:Blue; Integer) <span style="color:Blue; As <span style="color:Blue; Byte()
<span style="color:Blue; Set(<span style="color:Blue; ByVal value <span style="color:Blue; As <span style="color:Blue; Byte())
<span style="color:Blue; If value.<span style="color:Blue; Count - 1 > <span style="color:Blue; Order <span style="color:Blue; Then
<span style="color:Blue; If <span style="color:Blue; Not Ptrs.Keys.Contains(value(<span style="color:Blue; Order)) <span style="color:Blue; Then
Ptrs.Add(value(<span style="color:Blue; Order), <span style="color:Blue; New binarySignature)
<span style="color:Blue; End <span style="color:Blue; If
<span style="color:Blue; DirectCast(Ptrs(value(<span style="color:Blue; Order)), binarySignature).AddSignaturePtr(<span style="color:Blue; Order + 1) = value
<span style="color:Blue; Else
<span style="color:Blue; If <span style="color:Blue; Not Ptrs.Keys.Contains(value(<span style="color:Blue; Order)) <span style="color:Blue; Then
Ptrs.Add(value(<span style="color:Blue; Order), <span style="color:Blue; New Match)
<span style="color:Blue; End <span style="color:Blue; If
<span style="color:Blue; End <span style="color:Blue; If
<span style="color:Blue; End <span style="color:Blue; Set
<span style="color:Blue; End <span style="color:Blue; Property
<span style="color:Blue; End <span style="color:Blue; Class

<span style="color:Green; Next the scanning method:
<span style="color:Blue; Sub Scan(<span style="color:Blue; ByVal fls <span style="color:Blue; As <span style="color:Blue; Object, <span style="color:Blue; ByVal scantype <span style="color:Blue; As <span style="color:Blue; Integer)
<span style="color:Blue; If scantype = 0 <span style="color:Blue; Then
<span style="color:Blue; For <span style="color:Blue; Each f <span style="color:Blue; As <span style="color:Blue; String <span style="color:Blue; In <span style="color:Blue; DirectCast(fls, <span style="color:Blue; String())
<span style="color:Blue; Try
<span style="color:Blue; Dim BA() <span style="color:Blue; As <span style="color:Blue; Byte = IO.File.ReadAllBytes(f)
<span style="color:Green; Search starting at every byte position
<span style="color:Blue; For x = 0 <span style="color:Blue; To BA.<span style="color:Blue; Count
<span style="color:Green; Compare with the 60,000 signatures
BaseSignature.Search(BA, x)
<span style="color:Blue; Next
Threading.Interlocked.Increment(Cnt)
Threading.Interlocked.Increment(counts)
Threading.Interlocked.Add(Scanned, BA.<span style="color:Blue; Count)
<span style="color:Blue; Me.BeginInvoke(<span style="color:Blue; New Action(<span style="color:Blue; AddressOf UpdateCount))
<span style="color:Blue; Catch ex <span style="color:Blue; As Exception
<span style="color:Blue; End <span style="color:Blue; Try
<span style="color:Blue; Next
FlushMemory()
Cnt = 0
<span style="color:Blue; ElseIf scantype = 1 <span style="color:Blue; Then
<span style="color:Blue; Try
fileg = fls
<span style="color:Blue; Dim BA() <span style="color:Blue; As <span style="color:Blue; Byte = IO.File.ReadAllBytes(fls)
<span style="color:Green; Search starting at every byte position
<span style="color:Blue; For x = 0 <span style="color:Blue; To BA.<span style="color:Blue; Count
<span style="color:Green; Compare with the 60,000 signatures
BaseSignature.Search(BA, x)
<span style="color:Blue; Next
<span style="color:Blue; Me.BeginInvoke(<span style="color:Blue; New Action(<span style="color:Blue; AddressOf UpdateCount))
<span style="color:Blue; Catch ex <span style="color:Blue; As Exception
<span style="color:Blue; End <span style="color:Blue; Try
FlushMemory()
Cnt = 0
<span style="color:Blue; End <span style="color:Blue; If
<span style="color:Blue; End <span style="color:Blue; Sub
<span style="color:Blue; Private <span style="color:Blue; Sub UpdateCount()
<span style="color:Green; Label1.Text = CStr(counts) & " Files"
<span style="color:Green; Label1.Refresh()
<span style="color:Green; Label2.Text = Format(Scanned, "#,##0") & " Bytes"
<span style="color:Green; Label2.Refresh()
<span style="color:Blue; End <span style="color:Blue; Sub
<span style="color:Blue; Sub Found(<span style="color:Blue; ByVal Sender <span style="color:Blue; As <span style="color:Blue; Object, <span style="color:Blue; ByVal e <span style="color:Blue; As EventArgs)
<span style="color:Blue; Me.Invoke(<span style="color:Blue; New Action(<span style="color:Blue; AddressOf ReportFound))
<span style="color:Blue; End <span style="color:Blue; Sub
<span style="color:Blue; Public currentv <span style="color:Blue; As <span style="color:Blue; String
<span style="color:Blue; Sub ReportFound()
infected += 1
vnamecurrent = <span style="color:#A31515; "Malware" <span style="color:Green; I would like this to be the name of the signature that appears next to the byte array.
<span style="color:Blue; Dim x = fileg.LastIndexOf(<span style="color:#A31515; "")
<span style="color:Blue; Dim h = fileg.Substring(x + 1)
<span style="color:Blue; Dim b() = h.Split(<span style="color:#A31515; ".")
<span style="color:Blue; Dim jj = EndProcessAdvanced(b(0))
System.Threading.Thread.Sleep(10)
<span style="color:Blue; If jj = <span style="color:Blue; True <span style="color:Blue; Then
QuarantineFile(fileg)
<span style="color:Blue; ElseIf jj = <span style="color:Blue; False <span style="color:Blue; Then
QuarantineFile(fileg)
<span style="color:Blue; ElseIf jj = <span style="color:#A31515; "error" <span style="color:Blue; Then
log &= Now & <span style="color:#A31515; ": Error Occured while Killing Process " & b(0) & vbNewLine
<span style="color:Blue; End <span style="color:Blue; If
NotifyIcon1.BalloonTipIcon = ToolTipIcon.Warning
NotifyIcon1.BalloonTipTitle = <span style="color:#A31515; "Malware Detected and Quarantined!"
NotifyIcon1.BalloonTipText = <span style="color:#A31515; "Malware was detected in """ & fileg & <span style="color:#A31515; """" <span style="color:Green; """ & vnamecurrent & """
NotifyIcon1.ShowBalloonTip(10000)
<span style="color:Blue; Exit <span style="color:Blue; Sub
<span style="color:Blue; End <span style="color:Blue; Sub
[/code]
<div style="color:Black;background-color:White; <pre>
<span style="color:Green; And finally the adding the definition file to the dictionary
<span style="color:Blue; AddHandler binarySignature.MatchFound, <span style="color:Blue; AddressOf Found
<span style="color:Green; Open the file that contains the definitions
<span style="color:Blue; Dim readlinez() <span style="color:Blue; As <span style="color:Blue; String = System.IO.File.ReadAllLines(DefinitionFile)
<span style="color:Blue; For i = 0 <span style="color:Blue; To readlinez.<span style="color:Blue; Count - 1
<span style="color:Green; remove the text in front of the definition
<span style="color:Blue; If readlinez(i).Contains(<span style="color:#A31515; "*") <span style="color:Blue; Then
<span style="color:Blue; Else
<span style="color:Blue; Dim Def() <span style="color:Blue; As <span style="color:Blue; String = readlinez(i).Split(<span style="color:#A31515; "="c)
<span style="color:Blue; If Def(1).Contains(<span style="color:#A31515; "*") <span style="color:Blue; Then <span style="color:Blue; GoTo ex
<span style="color:Green; The byte array to hold the definition
<span style="color:Blue; Dim DefByte((Def(1).<span style="color:Blue; Count 2) - 1) <span style="color:Blue; As <span style="color:Blue; Byte
<span style="color:Blue; Try <span style="color:Green; < == This may remove any definitions with format error
<span style="color:Green; Transform the string definition in an array of byte
<span style="color:Blue; For x = 0 <span style="color:Blue; To Def(1).<span style="color:Blue; Count - 1 <span style="color:Blue; Step 2
<span style="color:Blue; Dim Bt <span style="color:Blue; As <span style="color:Blue; Byte = <span style="color:Blue; CByte(<span style="color:#A31515; "&H" & Def(1).Substring(x, 2))
DefByte(x 2) = Bt
<span style="color:Blue; Next
<span style="color:Green; Encode the tree
BaseSignature.AddSignaturePtr(0) = DefByte
<span style="color:Blue; Catch ex <span style="color:Blue; As Exception
<span style="color:Blue; End <span style="color:Blue; Try
<span style="color:Blue; End <span style="color:Blue; If
[/code]

<br/>
Secondly: I would like to somehow store the name of the virus (which is def(0) in the loading code) signature with the actual signature, so that when it finds a match, the name of the virus will be attached to the binary signature somehow, so I can identify
what one it is.
Thirdly: I would like to reduce memory usage. During the loading of the Dictionary, the average memory usage in KB is 800,000. That is just too much, I would like this prog to be able to run on a PC with 512MB of memory. During the Scan, The average memory
usage is about 600,000KB. Any way to reduce memory usage while being effective? (I can call Flush Memory but then the CPU gets greater.) I would just like the overall memory usage to be less than 100MB.
And Fourthly and last: I would like to DECREASE the CPU usage by a huge amount! Is there any way to limit a processs cpu usage, Kind of like Microsoft Security Essentials. It has the "Limit CPU usage during scan to: " and then up to 90%. I would like this
to be a maximum of 50% or even have a selection like MSE.<br/>
<br/>
Whoever responds to this post is a true SAINT and is greatly thanked by me!<br/>
Thank you if you can help me at all!
Zac<br/>
http://stykocompany.com/ <hr class="sig "When the going gets tough, the tough get Chuck Norris"

View the full article