Problems waiting for web browser control to finish loading page

[EDN Admin]

I have been tasked to do a project, and have been given a list of links from the client - what the Project needs to do is:
1 - open google with a specific search phrase - no problem
2 - look on that page for any links containing any member of the list of links, if found, invalidate the entry in the list, and auto-click the link, then go back
3 - repeat for any other "finds" on that page
4 - when done, auto-click on "Next" (no problem) and go back to step 2
For now, I am not even attempting to auto-click the found link and go back, I am just listing all found links in a TextBox. The problem I am having is waiting for the next pages to load before looking for a match in the list of links. I tried using webBrowser.DocumentCompleted
to set a Boolean - no help, I tried testing WebBrowserReadyState in the DocumentCompleted event - no help. So Far the code below gives the best results but it is inconsistent, meaning it completely misses some pages that I know have links on them that match.
As a test I also have code to just go to the Next" page and code to just look through one page at a time for the same links. This works perfectly and finds 6 to 8 links on all 10 pages I test every single time. Heres the code giving me fits, If anyone has
any ideas, Id sure appreciate it.
Happy Holidays.......

<div style="color:Black;background-color:White; <pre>
<span style="color:Blue; Private <span style="color:Blue; Sub btnFindLinks_Click(<span style="color:Blue; ByVal sender <span style="color:Blue; As System.Object, <span style="color:Blue; ByVal e <span style="color:Blue; As System.EventArgs) <span style="color:Blue; Handles btnFindLinks.Click
<span style="color:Blue; For page <span style="color:Blue; As <span style="color:Blue; Integer = 1 <span style="color:Blue; To 10 <span style="color:Green; go through first 11 pages - initial page already loaded
<span style="color:Blue; For <span style="color:Blue; Each element <span style="color:Blue; As HtmlElement <span style="color:Blue; In WebBrowser1.Document.Links <span style="color:Green; look at every link on that page
<span style="color:Blue; For Index <span style="color:Blue; As <span style="color:Blue; Integer = LBound(GoodLinks) <span style="color:Blue; To UBound(GoodLinks) <span style="color:Green; look through my list of desired links
<span style="color:Blue; If element.GetAttribute(<span style="color:#A31515; "HREF").Contains(GoodLinks(Index)) <span style="color:Blue; Then <span style="color:Green; if it is part of the link
GoodLinks(Index) = <span style="color:#A31515; "Already Used" <span style="color:Green; invalidate the desired link
TextBox1.AppendText(<span style="color:#A31515; "Found on Page # " & page.ToString & <span style="color:#A31515; " - " & element.GetAttribute(<span style="color:#A31515; "HREF") & vbCrLf)
<span style="color:Blue; End <span style="color:Blue; If
<span style="color:Blue; Next Index <span style="color:Green; go to next desired link
<span style="color:Blue; Next <span style="color:Green; go to next Web Page Link

<span style="color:Blue; For <span style="color:Blue; Each element2 <span style="color:Blue; As HtmlElement <span style="color:Blue; In WebBrowser1.Document.<span style="color:Blue; All <span style="color:Green; look through ALL elements
<span style="color:Blue; If element2.TagName = <span style="color:#A31515; "A" <span style="color:Blue; And element2.OuterText = <span style="color:#A31515; "Next" <span style="color:Blue; Then <span style="color:Green; find the Next Page link
element2.InvokeMember(<span style="color:#A31515; "click") <span style="color:Green; if found - click it
<span style="color:Blue; Exit <span style="color:Blue; For <span style="color:Green; and leave
<span style="color:Blue; End <span style="color:Blue; If
<span style="color:Blue; Next

<span style="color:Blue; Me.Cursor = Cursors.WaitCursor <span style="color:Green; show that we are busy
Threading.Thread.Sleep(1000) <span style="color:Green; Here is where the trouble starts - Wait a second
<span style="color:Blue; Do <span style="color:Green; Start Loop
Threading.Thread.Sleep(1000) <span style="color:Green; Wait more while waiting - what a concept
Application.DoEvents() <span style="color:Green; Yuck but doesnt work at all without this
<span style="color:Blue; Loop <span style="color:Blue; Until WebBrowser1.ReadyState = WebBrowserReadyState.Complete <span style="color:Green; wait till next page is complete
TextBox1.AppendText(<span style="color:#A31515; "Page # " & page.ToString & <span style="color:#A31515; " - " & WebBrowser1.ReadyState.ToString & vbCrLf)
<span style="color:Blue; Me.Cursor = Cursors.Arrow <span style="color:Green; restore normal cursor
<span style="color:Blue; Next page
<span style="color:Blue; End <span style="color:Blue; Sub
[/code]
<br/>
Lots of comments added as you can see

I also tried changing the sleep delay inside the loop to 100ms and even 50ms with no change. Also, the pages missed are not the same each time.


View the full article