HTML Screen Scraping using C# .Net WebClient

EDN Admin · Sep 23, 2012

I am trying the sample code from here:
http://www.codersource.net/MicrosoftNet/CAdvanced/HTMLScreenScrapinginC.aspx http://www.codersource.net/MicrosoftNet/CAdvanced/HTMLScreenScrapinginC.aspx
I have a Textbox1, Button1, and DataGrid1. Below is my code:
using System; 
using System.Collections.Generic; 
using System.ComponentModel; 
using System.Data; 
using System.Drawing; 
using System.Linq; 
using System.Text; 
using System.Windows.Forms; 
using System.Net; 
using System.IO; 
 
// creates a button protected 
using System.Web.UI.WebControls.Button Button1; 
// creates a byte array 
private byte[] aRequestHTML; 
// creates a string private string 
myString = null; 
// creates a datagrid protected 
using System.Web.UI.WebControls.DataGrid DataGrid1; 
// creates a textbox protected 
using System.Web.UI.WebControls.TextBox TextBox1; 
// creates the label protected 
using System.Web.UI.WebControls.Label Label1; 
// creates the arraylist 
private ArrayList a = new ArrayList();

namespace WindowsFormsApplication1 
{ 
public partial class Form1 : Form 
{ 
public Form1() 
{ 
InitializeComponent(); 
} 
 
private void Button1_Click(object sender, System.EventArgs e) 
{ 
// make an object of the WebClient class
 
WebClient objWebClient = new WebClient(); 
// gets the HTML from the url written in the textbox 
aRequestHTML = objWebClient.DownloadData(TextBox1.Text);
 
// creates UTf8 encoding object 
UTF8Encoding utf8 = new UTF8Encoding(); 
// gets the UTF8 encoding of all the html we got in aRequestHTML 
myString = utf8.GetString(aRequestHTML);
 
// this is a regular expression to check for the urls 
Regex r = new Regex("href\s*=\s*(?

?:\"(?[^\"]*)\")|(?[^\s]* ))");
 
// get all the matches depending upon the regular expression 
MatchCollection mcl = r.Matches(myString);
 
 
foreach(Match ml in mcl)
 
{ 
foreach(Group g in ml.Groups)
 
{ 
string b = g.Value + " "; 
// Add the extracted urls to the array list 
a.Add(b); 
 
} 
} 
// assign arraylist to the datasource 
DataGrid1.DataSource = a;
 
// binds the databind 
DataGrid1.DataBind();
 
 
// The following lines of code writes the extracted Urls to the file named test.txt 
StreamWriter sw = new StreamWriter(Server.MapPath("test.txt"));
 
sw.Write(myString);
 
sw.Close(); 
} 
 
 
} 
} 

Im guessing that there needs to be a reference to a WebService; not exactly sure how ot set that up, but I tried the following:
<img alt="" src="http://social.msdn.microsoft.com/Forums/getfile/169870

DataGridView Tasks > Choose Data Source > Add Project Data Source . . . > Service > Next > then, in the address bar I entered this: http:www.ebay.com

Then I got an error message:
<img alt="" src="http://social.msdn.microsoft.com/Forums/getfile/169871

Can someone please tell me what Im doing wrong here?
<
Ryan Shuell
 

View the full article

HTML Screen Scraping using C# .Net WebClient

EDN Admin

Well-known member

Similar threads