EDN Admin
Well-known member
Hello. I am writing my own specific product crawler. Now there is a product selling website which uses post data for pages. I really really need to able to post data and read the response. But they are using asp.net validation and it is so messed up. I really
could not figure how to properly post data and read. I am using htmlagilitypack. If it is possible to post data with htmlagilitypack and read the response it would be really really awesome.
Now this is the example page : http://www.hizlial.com/HizliListele.aspx?CatID=482643
http://www.hizlial.com/HizliListele.aspx?CatID=482643
When you opened the page look at the class "urun_listele"
You will see the options there <br/>
20 ÃrÃn Listele<br/>
40 ÃrÃn Listele<br/>
60 ÃrÃn Listele<br/>
TÃmÃnà Listele
Those numbers are product counts to be displayed. TÃmÃnà listele means list all products. Now i really need to post data and get all of the products under that product category. I used firebug to debug and tried to code below but i still got
default number of products
<div style="color:Black;background-color:White; <pre>
<span style="color:Blue; private <span style="color:Blue; void button11_Click(<span style="color:Blue; object sender, RoutedEventArgs e)
{
StringBuilder srBuilder = <span style="color:Blue; new StringBuilder();
AppendPostParameter(srBuilder, <span style="color:#A31515; "ctl00$ContentPlaceHolder1$cmbUrunSayi", <span style="color:#A31515; "full");
srBuilder = srBuilder.Replace(<span style="color:#A31515; "&", <span style="color:#A31515; "", srBuilder.Length - 1, 1);
<span style="color:Blue; byte[] byteArray = Encoding.UTF8.GetBytes(srBuilder.ToString());
HttpWebRequest hWebReq = (HttpWebRequest)WebRequest.Create(<span style="color:#A31515; "http://www.hizlial.com/HizliListele.aspx?CatID=482643");
hWebReq.Method = <span style="color:#A31515; "POST";
hWebReq.ContentType = <span style="color:#A31515; "application/x-www-form-urlencoded";
<span style="color:Blue; using (Stream requestStream = hWebReq.GetRequestStream())
{
requestStream.Write(byteArray, 0, byteArray.Length);
}
HtmlDocument hd = <span style="color:Blue; new HtmlDocument();
<span style="color:Blue; using (HttpWebResponse response = (HttpWebResponse)hWebReq.GetResponse())
{
<span style="color:Blue; using (StreamReader sr = <span style="color:Blue; new StreamReader(response.GetResponseStream()))
{
<span style="color:Blue; var htmlstring = sr.ReadToEnd();
}
}
}
<span style="color:Blue; static <span style="color:Blue; private <span style="color:Blue; void AppendPostParameter(StringBuilder sb, <span style="color:Blue; string name, <span style="color:Blue; string value)
{
sb.AppendFormat(<span style="color:#A31515; "{0}={1}&", name, HttpUtility.UrlEncode(value));
}
[/code]
After i get the data i will load it to the htmlagilitypack HtmlDocument
Any help is appreciated.
C# 4.0 , wpf application, htmlagiltiypack <hr class="sig Browser based http://www.monstermmorpg.com Pokemon Style MMORPG Game Developer Used asp.net 4.0 routing at its
http://www.monstermmorpg.com/MonsterDex Monsters
View the full article
could not figure how to properly post data and read. I am using htmlagilitypack. If it is possible to post data with htmlagilitypack and read the response it would be really really awesome.
Now this is the example page : http://www.hizlial.com/HizliListele.aspx?CatID=482643
http://www.hizlial.com/HizliListele.aspx?CatID=482643
When you opened the page look at the class "urun_listele"
You will see the options there <br/>
20 ÃrÃn Listele<br/>
40 ÃrÃn Listele<br/>
60 ÃrÃn Listele<br/>
TÃmÃnà Listele
Those numbers are product counts to be displayed. TÃmÃnà listele means list all products. Now i really need to post data and get all of the products under that product category. I used firebug to debug and tried to code below but i still got
default number of products
<div style="color:Black;background-color:White; <pre>
<span style="color:Blue; private <span style="color:Blue; void button11_Click(<span style="color:Blue; object sender, RoutedEventArgs e)
{
StringBuilder srBuilder = <span style="color:Blue; new StringBuilder();
AppendPostParameter(srBuilder, <span style="color:#A31515; "ctl00$ContentPlaceHolder1$cmbUrunSayi", <span style="color:#A31515; "full");
srBuilder = srBuilder.Replace(<span style="color:#A31515; "&", <span style="color:#A31515; "", srBuilder.Length - 1, 1);
<span style="color:Blue; byte[] byteArray = Encoding.UTF8.GetBytes(srBuilder.ToString());
HttpWebRequest hWebReq = (HttpWebRequest)WebRequest.Create(<span style="color:#A31515; "http://www.hizlial.com/HizliListele.aspx?CatID=482643");
hWebReq.Method = <span style="color:#A31515; "POST";
hWebReq.ContentType = <span style="color:#A31515; "application/x-www-form-urlencoded";
<span style="color:Blue; using (Stream requestStream = hWebReq.GetRequestStream())
{
requestStream.Write(byteArray, 0, byteArray.Length);
}
HtmlDocument hd = <span style="color:Blue; new HtmlDocument();
<span style="color:Blue; using (HttpWebResponse response = (HttpWebResponse)hWebReq.GetResponse())
{
<span style="color:Blue; using (StreamReader sr = <span style="color:Blue; new StreamReader(response.GetResponseStream()))
{
<span style="color:Blue; var htmlstring = sr.ReadToEnd();
}
}
}
<span style="color:Blue; static <span style="color:Blue; private <span style="color:Blue; void AppendPostParameter(StringBuilder sb, <span style="color:Blue; string name, <span style="color:Blue; string value)
{
sb.AppendFormat(<span style="color:#A31515; "{0}={1}&", name, HttpUtility.UrlEncode(value));
}
[/code]
After i get the data i will load it to the htmlagilitypack HtmlDocument
Any help is appreciated.
C# 4.0 , wpf application, htmlagiltiypack <hr class="sig Browser based http://www.monstermmorpg.com Pokemon Style MMORPG Game Developer Used asp.net 4.0 routing at its
http://www.monstermmorpg.com/MonsterDex Monsters
View the full article