EDN Admin
Well-known member
I am re-posing this question because the prior post has gotten really full of side issues so I fear the main problems has been lost.
I have a windows forms application written in VB.NET which scrapes information from serveral web pages? I am having problem with one particular site where partway through the page navigation my application hangs. When I press pause (or break)
in the debugger, it stops on a call to System.Windows.Forms.Application.DoEvents.
Resuming execution shows that it really is stuck on this line (it does not reach the next line of code). It also hangs about the same point each time I run it (at least its consistent). Does anyone have a clue why this would happen? Why would
any call to DoEvents hang?
Here is the code along with an explanation of what I am doing. First, I have a control class that wraps a web browser control:
Public Class WebBrowserEx
Inherits WebBrowser
Next, I added a function to navigate to a page and to click a page element. Both of these actions cause a new page to load and require the calling program to wait until the page is loaded.
<pre> Public Shadows Function Navigate(ByVal url As System.Uri, _
Optional ByVal timeoutSeconds As Integer = 60) As Boolean
Initialize the page ready detection
Call InitializePageReadyDetection()
Tell the browser to load the web page at the given URL
MyBase.Navigate(url)
Wait for the browser to finish loading the current page.
Return WaitUntilPageReady(timeoutSeconds)
End Function
Public Function ClickPageElement(ByVal pageElement As mshtml.IHTMLElement,
Optional ByVal timeoutSeconds As Integer = 60) As Boolean
Initialize the page ready detection
Call InitializePageReadyDetection()
Click the element. WaitForBrowserControl doesnt work after a element click
So all we can do is wait for a fixed period of time.
pageElement.click()
Wait for the browser to finish loading the current page.
Return WaitUntilPageReady(timeoutSeconds)
End Function
[/code]
Here is the Initialize page ready function:
<pre> Private Sub InitializePageReadyDetection()
mPageDocumentCompletedEventOccured = False
End Sub
[/code]
where mPageDocumentCompletedEventOccured is a private, class level boolean
Here is the function that waits for the page to be completed. The Doevents call here is where it is getting hung up.
<pre> Waits for the current page to be completely loaded. Returns True if it is and false if it failed to load.
Private Function WaitUntilPageReady(Optional ByVal timoutSeconds As Integer = 60) As Boolean
Dim startTime As Date
Dim stopTime As Date
Dim secondsElapsed As Long
Setup error handling
WaitUntilPageReady = False
Wait for the browser to finish loading the page. The page is finished when BOTH of the following are true:
1. The document completed event for the entire page has occured (see webCtl_DocumentCompleted() below)
2. The browser control reports that the page is ready.
There are problems using one or the other, so both must be used.
secondsElapsed = 0
startTime = Now
Do While (mPageDocumentCompletedEventOccured = False OrElse MyBase.ReadyState <> System.Windows.Forms.WebBrowserReadyState.Complete) AndAlso secondsElapsed < timoutSeconds
stopTime = Now
secondsElapsed = DateDiff(Microsoft.VisualBasic.DateInterval.Second, startTime, stopTime)
System.Threading.Thread.Sleep(100) sleep method takes time in milliseconds
System.Windows.Forms.Application.DoEvents ()
Loop
If we document completed event for the entire page has occured and the browser also reports that is is ready, then the page is ready.
If mPageDocumentCompletedEventOccured = True AndAlso MyBase.ReadyState = System.Windows.Forms.WebBrowserReadyState.Complete Then
WaitUntilPageReady = True
If we did not handle the original URL document completed event (because it did not get raised) and the browsers ready state is
Interactive or Complete, report that the page is ready anyway. The state of interactive means that enough of the page is
available for user interaction to occur on a limited basis.
ElseIf mPageDocumentCompletedEventOccured = False AndAlso (MyBase.ReadyState = System.Windows.Forms.WebBrowserReadyState.Interactive Or MyBase.ReadyState = System.Windows.Forms.WebBrowserReadyState.Complete) Then
WaitUntilPageReady = True
If the page is not ready in the alloted time, stop the navigation.
Else
MyBase.Stop()
End If
End Function
[/code]
The last piece of the puzzle is the document completed event handler of the web control. While the routine above is looping, the document completed event will eventually get called and set the mPageDocumentCompletedEventOccured event to true which
will allow the wait function above to exit the loop.
<pre> Uses the document completed event to know when the page requested has been fully loaded and ready for interaction.
Private Sub webCtl_DocumentCompleted(ByVal eventSender As System.Object, ByVal eventArgs As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles MyBase.DocumentCompleted
This event can fire more than once per page. It will fire once for each sub-document contained within a page
(like a frame for example) and then again for the page itself. The eventArgs contains the URL of the (sub)document
that has finished loading. The entire page isnt ready until the event for the entire page has been raised.
It is possible to get additional document completed events AFTER the document completed event from the page fires.
This happens when tracking sites are used (e.g. shareit) and the events dont fire in the order expected. The
following if check will make sure those additional events dont screw up our logic.
If mPageDocumentCompletedEventOccured = False Then
mPageDocumentCompletedEventOccured = eventArgs.Url.Equals(MyBase.Url)
End If
End Sub[/code]
This entire function works on all but one site. Even on that site, it works for about 13 pages and then hangs. Not sure why. Any ideas why DoEvents would not return after getting part way?
<hr class="sig Andy
View the full article
I have a windows forms application written in VB.NET which scrapes information from serveral web pages? I am having problem with one particular site where partway through the page navigation my application hangs. When I press pause (or break)
in the debugger, it stops on a call to System.Windows.Forms.Application.DoEvents.
Resuming execution shows that it really is stuck on this line (it does not reach the next line of code). It also hangs about the same point each time I run it (at least its consistent). Does anyone have a clue why this would happen? Why would
any call to DoEvents hang?
Here is the code along with an explanation of what I am doing. First, I have a control class that wraps a web browser control:
Public Class WebBrowserEx
Inherits WebBrowser
Next, I added a function to navigate to a page and to click a page element. Both of these actions cause a new page to load and require the calling program to wait until the page is loaded.
<pre> Public Shadows Function Navigate(ByVal url As System.Uri, _
Optional ByVal timeoutSeconds As Integer = 60) As Boolean
Initialize the page ready detection
Call InitializePageReadyDetection()
Tell the browser to load the web page at the given URL
MyBase.Navigate(url)
Wait for the browser to finish loading the current page.
Return WaitUntilPageReady(timeoutSeconds)
End Function
Public Function ClickPageElement(ByVal pageElement As mshtml.IHTMLElement,
Optional ByVal timeoutSeconds As Integer = 60) As Boolean
Initialize the page ready detection
Call InitializePageReadyDetection()
Click the element. WaitForBrowserControl doesnt work after a element click
So all we can do is wait for a fixed period of time.
pageElement.click()
Wait for the browser to finish loading the current page.
Return WaitUntilPageReady(timeoutSeconds)
End Function
[/code]
Here is the Initialize page ready function:
<pre> Private Sub InitializePageReadyDetection()
mPageDocumentCompletedEventOccured = False
End Sub
[/code]
where mPageDocumentCompletedEventOccured is a private, class level boolean
Here is the function that waits for the page to be completed. The Doevents call here is where it is getting hung up.
<pre> Waits for the current page to be completely loaded. Returns True if it is and false if it failed to load.
Private Function WaitUntilPageReady(Optional ByVal timoutSeconds As Integer = 60) As Boolean
Dim startTime As Date
Dim stopTime As Date
Dim secondsElapsed As Long
Setup error handling
WaitUntilPageReady = False
Wait for the browser to finish loading the page. The page is finished when BOTH of the following are true:
1. The document completed event for the entire page has occured (see webCtl_DocumentCompleted() below)
2. The browser control reports that the page is ready.
There are problems using one or the other, so both must be used.
secondsElapsed = 0
startTime = Now
Do While (mPageDocumentCompletedEventOccured = False OrElse MyBase.ReadyState <> System.Windows.Forms.WebBrowserReadyState.Complete) AndAlso secondsElapsed < timoutSeconds
stopTime = Now
secondsElapsed = DateDiff(Microsoft.VisualBasic.DateInterval.Second, startTime, stopTime)
System.Threading.Thread.Sleep(100) sleep method takes time in milliseconds
System.Windows.Forms.Application.DoEvents ()
Loop
If we document completed event for the entire page has occured and the browser also reports that is is ready, then the page is ready.
If mPageDocumentCompletedEventOccured = True AndAlso MyBase.ReadyState = System.Windows.Forms.WebBrowserReadyState.Complete Then
WaitUntilPageReady = True
If we did not handle the original URL document completed event (because it did not get raised) and the browsers ready state is
Interactive or Complete, report that the page is ready anyway. The state of interactive means that enough of the page is
available for user interaction to occur on a limited basis.
ElseIf mPageDocumentCompletedEventOccured = False AndAlso (MyBase.ReadyState = System.Windows.Forms.WebBrowserReadyState.Interactive Or MyBase.ReadyState = System.Windows.Forms.WebBrowserReadyState.Complete) Then
WaitUntilPageReady = True
If the page is not ready in the alloted time, stop the navigation.
Else
MyBase.Stop()
End If
End Function
[/code]
The last piece of the puzzle is the document completed event handler of the web control. While the routine above is looping, the document completed event will eventually get called and set the mPageDocumentCompletedEventOccured event to true which
will allow the wait function above to exit the loop.
<pre> Uses the document completed event to know when the page requested has been fully loaded and ready for interaction.
Private Sub webCtl_DocumentCompleted(ByVal eventSender As System.Object, ByVal eventArgs As System.Windows.Forms.WebBrowserDocumentCompletedEventArgs) Handles MyBase.DocumentCompleted
This event can fire more than once per page. It will fire once for each sub-document contained within a page
(like a frame for example) and then again for the page itself. The eventArgs contains the URL of the (sub)document
that has finished loading. The entire page isnt ready until the event for the entire page has been raised.
It is possible to get additional document completed events AFTER the document completed event from the page fires.
This happens when tracking sites are used (e.g. shareit) and the events dont fire in the order expected. The
following if check will make sure those additional events dont screw up our logic.
If mPageDocumentCompletedEventOccured = False Then
mPageDocumentCompletedEventOccured = eventArgs.Url.Equals(MyBase.Url)
End If
End Sub[/code]
This entire function works on all but one site. Even on that site, it works for about 13 pages and then hangs. Not sure why. Any ideas why DoEvents would not return after getting part way?
<hr class="sig Andy
View the full article