G
Gidmaestro
Guest
I coded a program that lets me download a set of links that I paste into a textbox. I could do the same thing by visiting each page in my browser and saving it manually, but this speeds things up and is a convenience.
One problem I noticed is when looking at a forum page such as:
2D Object Recognition Project
I find that in a browser, the whole page doesn't 'come at once - it seems to be buffered, and when I scroll down, there is a pause before the next section loads. It is as if very long pages, or pages that are generated by forum software, behave differently than ordinary web pages in Microsoft Edge. This is also true if I attempt to load the page via by VB.net page using code such as:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls Or SecurityProtocolType.Tls11 Or SecurityProtocolType.Tls12
' Create a request for the URL.
Dim request As WebRequest = WebRequest.Create(strURL)
' If required by the server, set the credentials.
request.Credentials = CredentialCache.DefaultCredentials
' Get the response.
Dim response As HttpWebResponse = CType(request.GetResponse(), HttpWebResponse)
' Display the status.
' Get the stream containing content returned by the server.
Dim dataStream As Stream = response.GetResponseStream()
' Open the stream using a StreamReader for easy access.
Dim reader As New StreamReader(dataStream)
' Read the content.
If shouldLimitPageSize Then
Dim charbuf(ClassGlobalVariables.pageSizeLimit + 2) As Char
Dim i As Integer
For i = 0 To charbuf.Length - 1
charbuf(i) = " "
Next
reader.ReadBlock(charbuf, 0, ClassGlobalVariables.pageSizeLimit)
responseFromServer = New String(charbuf)
Else
responseFromServer = reader.ReadToEnd()
End If
I seem to not be getting the entire page, because I see more content when I view manually in my browser, patiently scrolling down each time to see more, than if I load it via a routine in VB.net for later offline viewing. Is there a way to cope with very long forum type pages?
Thanks.
Continue reading...
One problem I noticed is when looking at a forum page such as:
2D Object Recognition Project
I find that in a browser, the whole page doesn't 'come at once - it seems to be buffered, and when I scroll down, there is a pause before the next section loads. It is as if very long pages, or pages that are generated by forum software, behave differently than ordinary web pages in Microsoft Edge. This is also true if I attempt to load the page via by VB.net page using code such as:
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls Or SecurityProtocolType.Tls11 Or SecurityProtocolType.Tls12
' Create a request for the URL.
Dim request As WebRequest = WebRequest.Create(strURL)
' If required by the server, set the credentials.
request.Credentials = CredentialCache.DefaultCredentials
' Get the response.
Dim response As HttpWebResponse = CType(request.GetResponse(), HttpWebResponse)
' Display the status.
' Get the stream containing content returned by the server.
Dim dataStream As Stream = response.GetResponseStream()
' Open the stream using a StreamReader for easy access.
Dim reader As New StreamReader(dataStream)
' Read the content.
If shouldLimitPageSize Then
Dim charbuf(ClassGlobalVariables.pageSizeLimit + 2) As Char
Dim i As Integer
For i = 0 To charbuf.Length - 1
charbuf(i) = " "
Next
reader.ReadBlock(charbuf, 0, ClassGlobalVariables.pageSizeLimit)
responseFromServer = New String(charbuf)
Else
responseFromServer = reader.ReadToEnd()
End If
I seem to not be getting the entire page, because I see more content when I view manually in my browser, patiently scrolling down each time to see more, than if I load it via a routine in VB.net for later offline viewing. Is there a way to cope with very long forum type pages?
Thanks.
Continue reading...