POSTing data with HttpWebRequest - problems connecting

akm · Dec 12, 2003

Im making a program that attempts to POST form data to the All Music Guide site, get the servers HTML response, and then parse it to extract various bits of information. I think I understand what I have to do, but Im having trouble getting it to work.

Basically, I use a HttpWebRequest to connect to the AMG site, POST the data, and then get the response stream and read in the response. However, I randomly get one of two errors - first, I sometimes get a WebException from HttpWebRequests CheckFinalStatus() method, with message "The underlying connection was closed - an unexpected error occured on a receive".

Ive searched the net for this particular exception, and the two solutions Ive seen are modifying the KeepAlive property (setting it to false), and changing the .Net configuration file "machine.config" to not use the default proxy. However, neither of these suggestions seems to have helped. In fact, changing the machine.config file to not use the system default proxy settings makes the program fail 100% of the time (my IE connection settings do not have the box "Use a proxy server" ticked).

Another problem, which occurs less frequently, is that I sometimes get back a HTML page from the server looking something like this:
<HTML><HEAD><META HTTP-EQUIV="Refresh" CONTENT=0; URL=http://www.allmusic.com/cg/amg.dll></HEAD><BODY></BODY></HTML>

The URL is the DLL to which Im sending the form data. So does this mean that Im supposed to repost the data? I understand that on a browser this would attempt to immediately refresh to the URL given, but does this mean that the browser must repost the data? Im not sure whether this is standard behaviour or whether theres something wrong (because I would imagine that the server would choose to wait until it gave me proper data, instead of making me redo my query).

Heres what my code looks like:

Code:

HttpWebRequest webRequest = null;

webRequest = (HttpWebRequest) WebRequest.Create (AMG_QUALIFIED_DLL); 

webRequest.KeepAlive = false;
webRequest.Timeout = 100000;
webRequest.Method = "POST";
webRequest.ContentType = "application/x-www-form-urlencoded";

// Convert the postData into a byte array
// postData is an instance variable
byte[] requestBytes = (new System.Text.ASCIIEncoding()).GetBytes (postData);
webRequest.ContentLength = requestBytes.Length;

// Write the bytes to the request stream
Stream requestStream = webRequest.GetRequestStream();
requestStream.Write (requestBytes, 0, requestBytes.Length);
requestStream.Close();

HttpWebResponse webResponse = null;
StreamReader sr = null;

try
{
    webResponse = (HttpWebResponse) webRequest.GetResponse(); 

    sr = new StreamReader (webResponse.GetResponseStream(), System.Text.Encoding.ASCII);
    response = sr.ReadToEnd ();
}

catch (WebException ex)
{
    Console.WriteLine (ex.Message + ex.StackTrace);
    throw new WebException (ex.Message, ex.InnerException, ex.Status, ex.Response);
}

finally
{
    if (sr != null)
        sr.Close(); 
    if (webResponse != null)
        webResponse.Close ();

    sr = null;
    webResponse = null;
}

Now, what really puzzles me is this - if I use the ActiveX WebBrowser control to POST the data, it seems to work fine 100% of the time! It would appear then that there is something wrong with what Im doing, but Im not sure what.

Heres the WebBrowser code (its just a slightly modified version of the MS sample code) :

Code:

object oEmpty = "";
string cPostData = "p=amg&sql="+artistNameBox.Text+"&opt1=1";
object vHeaders = "Content-Type: application/x-www-form-urlencoded" + "\n" + "\r";
object vPost = System.Text.ASCIIEncoding.ASCII.GetBytes(cPostData);

this.axWebBrowser1.Navigate ("http://www.allmusic.com/cg/amg.dll", ref oEmpty, ref oEmpty, ref vPost, ref vHeaders);

I though that perhaps this had to do with the UserAgent property of the request, and so I tried setting it to the string for IE 6 (I cant remember the string, something like "Mozilla 4.0; compatible IE6.0"?). However, that didnt seem to change things either.

Any help would be much appreciated!

css145hs · Jan 24, 2004

hey, did you ever find a resolution to your problem, I would be greatly interested if you did

karlhaak · Feb 2, 2004

connection closed on webrequest

Thank you very much for your post here.
The option with keepAlive=false works excellent for me. I was experiencing the following behavior.

I was writing a windows service that would connect to a website, post some data to get a response. Everything worked fine when I ran the component as a console Application in Debug mode on my machine . Once installed as a service every second request failed.

I had to fight with an unvalid certificate for ssl, username, password and a proxyserver which gave me plenty of options to look for.

I finally tried the keepAlive option and everything works fine. I still do not understand the influence of this parameter on the failure of the connection every second time and would be interested in any ideas.

The environment I am working in is Windows 2000 with .NET 1.0.

I created an abstract of my solution as a coding example. Hope this will safe some time for somebody else.

Karl

Code:

Imports System.IO
Imports System.Text
Imports System.Threading
Imports System.Net
Imports System.Text.Encoding
Imports System.Security.Cryptography.X509Certificates

Public Class MyCertificateValidation
    Implements System.Net.ICertificatePolicy

    This class handles problems with certificates if ssl (https) is used

    Public Function CheckValidationResult(ByVal srvPoint As ServicePoint, _
    ByVal cert As X509Certificate, ByVal request As WebRequest, ByVal problem As Integer) _
       As Boolean Implements ICertificatePolicy.CheckValidationResult
        Return True  Accept all certificates
    End Function
End Class

Public Class RequestState
    Public request As WebRequest = Nothing
    Public requestDocument As String
End Class

Public Class HTMLRequest

    Handler to synchronise with main thread again
    Public allDone As New ManualResetEvent(False)

    Private _requestDocument As String = "Text to send"
    Private _responseDocument As String
    Private _web As Net.HttpWebRequest

    Private Sub ReadCallback(ByVal asynchronousResult As IAsyncResult)
        Get the information to pass on from the Async Object
        Dim myRequestState As RequestState = CType(asynchronousResult.AsyncState, RequestState)
        Dim myWebRequest2 As WebRequest = myRequestState.request
        Dim requestDocument As String = myRequestState.requestDocument

        Get the stream to write the request to
        Dim streamResponse As Stream = myWebRequest2.EndGetRequestStream(asynchronousResult)
        Dim bytArray() As Byte = System.Text.Encoding.ASCII.GetBytes(requestDocument)
        streamResponse.Write(bytArray, 0, requestDocument.Length)
        streamResponse.Close()

        Tell the main thread that writing to stream is complete
        allDone.Set()
    End Sub

    Public Sub retrieve()
        Dim bytOutArray() As Byte
        Dim objWebResponse As Net.WebResponse = _web.GetResponse
        Dim objStream As IO.StreamReader
        objStream = New IO.StreamReader(objWebResponse.GetResponseStream(), System.Text.ASCIIEncoding.ASCII)
        _responseDocument = objStream.ReadToEnd()
        close all streams
        objWebResponse.Close()
        objStream.Close()
    End Sub

    Public Sub send()
        Prepare Object to pass into asynchron call
        Dim myRequestState As New RequestState
        myRequestState.request = _web
        myRequestState.requestDocument = _requestDocument
        Dim r As IAsyncResult = CType(_web.BeginGetRequestStream(AddressOf ReadCallback, myRequestState), IAsyncResult)

        Wait till asychrone is done ## allDone.Set() ##
        allDone.WaitOne()
    End Sub

    Public Sub getRequest()

        Configure connection

        in case of problem with certificate
        ServicePointManager.CertificatePolicy = New MyCertificateValidation

        Create request object
        _web = CType(WebRequest.Create("https://www.somedomain.com"), HttpWebRequest)
        With _web
            .ContentType = "application/x-www-form-urlencoded"
            .Method = "POST"
            .KeepAlive = False
            .ContentLength = _requestDocument.Length
            .Timeout = 10000

            in case of proxyserver
            .Proxy = New WebProxy("xxx.proxyserver.com", 8080)

            in case of required password
            Dim cache As CredentialCache = New CredentialCache
            cache.Add(New Uri("https://www.somedomain.com/"), "Basic", New NetworkCredential("Username", "password"))
            .Credentials = cache
        End With

        send() send the request
        retrieve() get the data
    End Sub
End Class

karlhaak · Feb 3, 2004

connection closed on webrequest Microsoft article

I finally found the Microsoft article that applies to the problem.
It can be found under the following location

http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B819450

There is also some good advice in
Jan Tielens Blog

Hope this will help someone who stumbles over the same problem.

Karl

aric · Jul 29, 2004

I have hit the same problem...

I am trying to implement a financial gateway system, which uses WSE 1.0 SP1, and the sample code provided makes an SSL connection to their server. So, I have to find a way to override the System.Net.ICertificatePolicy so as to always approve SSL certs.

I am running ASP.NET (VB) .NET version 1.1, and am having a hard time getting your code to work, and was wondering if you had any suggestions.

I put your class above my class (in a code-behind page which inherits the page) thusly:

Code:

Public Class MyCertificateValidation
    Implements System.Net.ICertificatePolicy

    This class handles problems with certificates if ssl (https) is used

    Public Function CheckValidationResult(ByVal srvPoint As ServicePoint, _
    ByVal cert As X509Certificate, ByVal request As WebRequest, ByVal problem As Integer) _
       As Boolean Implements ICertificatePolicy.CheckValidationResult
        Return True  Accept all certificates
    End Function
End Class

Public Class thepage

     Inherit the page
    Inherits Page
.
.
.

Everything looks good, but I get this error:

Code:

Compilation Error

Description: An error occurred during the compilation of a resource required to service this request. Please review the following specific error details and modify your source code appropriately.

Compiler Error Message: BC30149: MyCertificateValidation must implement Overridable Function CheckValidationResult(srvPoint As ServicePoint, certificate As Security.Cryptography.X509Certificates.X509Certificate, request As WebRequest, certificateProblem As Integer) As Boolean for interface System.Net.ICertificatePolicy.

Source Error:

Line 26: 
Line 27: Public Class MyCertificateValidation
Line 28:     Implements System.Net.ICertificatePolicy
Line 29: 
Line 30:     This class handles problems with certificates if ssl (https) is used

However, it clearly DOES overright it... and I am stumped.

Any ideas anyone?

Thanks in advance.

karlhaak · Jul 30, 2004

Quick suggestions for: Compiler Error Message: BC30149

Hello Aric,

I cannot see any problem at first glance. Here are some quick things I would like to check.

1) If you programm with visual studio, it will mark the class as invalid if not all declared interfaces have been implemented. Can you check that.

2) Put the class in an extra file.

3) check if the namespace
Imports System.Security.Cryptography.X509Certificates
has been imported.

4) check if the namespace
Imports System.Net
has been imported.

hope one of these will help

Karl

mary92028 · Jan 23, 2008

Karl - you and Jan helped me with my SSL problem in VB.net. MUCH APPRECIATED.

Mary

POSTing data with HttpWebRequest - problems connecting

akm

New member

css145hs

New member

karlhaak

Member

karlhaak

Member

aric

New member

karlhaak

Member

mary92028

New member

Similar threads