HttpWebRequest timing out?

[krinpit]

Hi,

I need to read and parse a webpage to obtain certain information from it on a daily basis, in a VB.NET environment. I can successfully initialise the request and can receive some information from the webpage in question, but its as if it sometimes times out and only returns to me what I managed to obtain from the stream while it was alive. Sometimes I manage to get only around 500 of the lines into my arraylist at the end. Please see my code and see if Im missing something...

        Dim arrBuff As New ArrayList()  Results go in here
        Dim myHttpWebRequest As HttpWebRequest
        Dim myHttpWebResponse As HttpWebResponse
        Dim receiveStream As Stream
        Dim encode As Encoding
        Dim sr As IO.StreamReader

        Try
            myHttpWebRequest = CType(WebRequest.Create(strURL), HttpWebRequest)
            myHttpWebResponse = CType(myHttpWebRequest.GetResponse(), HttpWebResponse)
            receiveStream = myHttpWebResponse.GetResponseStream()
            encode = System.Text.Encoding.GetEncoding("utf-8")
            sr = New StreamReader(receiveStream, encode)

            Do Until sr.Peek = -1
                strLine = sr.ReadLine
                arrBuff.Add(strLine)
            Loop
        Catch ex As System.Net.WebException
            MsgBox(ex.Message)
        Finally
            sr.Close()
            myHttpWebResponse.Close()
        End Try

 

[krinpit]

Well, the good news is that Ive solved my problem. (Ive posted the new code below).

The bad news is, I had to use an ugly work around, and it doesnt diminish my curiousity as to why the previous method didnt work.

The URL in question is:

http://www.msci.com/eqpages/body_DM.S.8.html

    Public Function readWebPage(ByVal strURL As String) As ArrayList

        Dim arrBuff As ArrayList
        Dim sr As IO.StreamReader

        Dim hwRequest As HttpWebRequest
        Dim hwResponse As HttpWebResponse
        Dim receiveStream As Stream

        Try
            hwRequest = CType(WebRequest.Create(strURL), HttpWebRequest)
            hwRequest.Timeout = 999999999
            hwResponse = CType(hwRequest.GetResponse(), HttpWebResponse)
            receiveStream = hwResponse.GetResponseStream()
            sr = New StreamReader(receiveStream)

            Dim sLongStr As String = sr.ReadToEnd() Read stream into a string, all in one go
            arrBuff = MultiLineStringToArrayList(sLongStr)
        Catch ex As System.Net.WebException
            MsgBox(ex.Message)
        Catch ex As Exception
            MsgBox(ex.Message)
        Finally
            If Not (sr Is Nothing) Then sr.Close()
            If Not (hwResponse Is Nothing) Then hwResponse.Close()
            hwRequest = Nothing
        End Try

        Return arrBuff
    End Function

    Private Function MultiLineStringToArrayList(ByVal sLongStr As String) As ArrayList
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        This Function divides a string into a new string for every line
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        sLongStr = sLongStr.Replace(vbCrLf, vbLf)  Replace any Carriage returns with a normal LineFeed
        Return New ArrayList(sLongStr.Split(vbLf))   Array with a line per element
    End Function