Performance question VB .NET /C/C++/VB6/C#

[vokuit00]

Hi,

i have a general question about a project that has just begun. I should read lines out of a file. A line (in ASCII format) looks like this:

timestamp(Double) channel(Byte) ID(Integer) mode(String with 2 characters) lenght(Byte) data(max. 8 Byte)

I should read in this lines, parse and visualize them.

My first trials with a file of 60.000 lines were a bit disillusioning. I tried to visualize them in a listbox-control and it needs 10 min. to load and visualize them. The next trial with a listview-control was about one minute. But yesterday i got the information that the maximum file size was 1,500,000 lines.

Now my naive question is, how to handle such large amount of data in VB .NET? Or should i use C/C++ for a better performance?

My first thougts were to implement a class in VB .NET in which i "fill" in the data after parsing the lines with RegEx. After that i could put this class in an Array or a Collection or a database???

I must visualize this data on three kinds. First as a kind of treeview, second in an ActiveX-component like an oszilloskope and third in a table. The user must manipulate some parts of this data.

I hope you can excuse my bad English and that you can give me some recommendation for working on this large amount of data.

Many Thanks

Volker

 

[Denaes]

Post the code youre using to read the ASCII data.

.Net provides some easy ways to access ASCII, which by default are slower. So unless youve looked into better ASCII performance - youre not getting it. It doesnt help that most VB users are still able to use the old VB6 function which is really slow.

Post your code and someone can advise you on how to kick it up a notch.

 

[vokuit00]

Some kind of code

Hi,

this morning i tried some tests:

Dim start As Integer
Dim ende As Integer
Dim anzahl As Integer
Dim tmpString As String
start = System.Environment.TickCount()
ListView1.BeginUpdate()
While (readstream.Peek() <> -1)
tmpString = readstream.ReadLine()
ListView1.Items.Add(tmpString)
anzahl += 1
End While
ListView1.EndUpdate()
ende = System.Environment.TickCount()
MessageBox.Show("Es dauerte: " & CType(ende - start, String) & " f

 

[Denaes]

Dont have VB.Net available on this machine, but I can see one area or two to trim some fat.

Unless you post the code with your connection ("readstream") people cannot look into helping with that.


As it is, you can remove the string tmpString. Unless you cut out code where it does something else - this is a bottleneck. Not saying its your problem, but Strings are fairly slow and youre copying it an extra time here.

Dim start As Integer
      Dim ende As Integer
      Dim anzahl As Integer
      Dim tmpString As String
      start = System.Environment.TickCount()
      ListView1.BeginUpdate()
      While (readstream.Peek() <> -1)
         tmpString = readstream.ReadLine()
         ListView1.Items.Add(tmpString)
ListView1.Items.Add(readstream.ReadLine())
         anzahl += 1
      End While
      ListView1.EndUpdate()
      ende = System.Environment.TickCount()
      MessageBox.Show("Es dauerte: " & CType(ende - start, String) & " f

 

[vokuit00]

Answer:

Is use exactly the same code for the ListBox. Only replace ListView1 with ListBox1.

Thanks

Volker

 

[penfold69]

If the ascii data is in a relatively simple format of CSV or Tab Delimited, then you may be able to make use of the ODBC driver, and fill a datatable with it.

I doubt if it will actually result in much of an ACTUAL improved speed, but it might make it more manageable to thread it, so that it is perceived to be faster.

Ive worked on importing and exporting CSV/TAB files a bit recently, and Ive found the ODBC driver is the most flexible to work with.

Of course, if your data is in true binary form, then it probably wont work for you.

B.

 

[Nerseus]

Does the listbox have sorting turned on? That might account for its extreme slowness. If youre retrieving from a database, put the "ORDER BY" there and turn sorting off in your listbox/combobox to save time.

I cant see any reason to fill a listbox with more than a few dozen entries, maybe a few hundred if really needed. Anything more and you need a different solution.

-ner

 

[Denaes]

I cant see any reason to fill a listbox with more than a few dozen entries, maybe a few hundred if really needed. Anything more and you need a different solution.

-ner

What? You dont want to scroll through 1.5 million items?

Correct me if Im wrong, wouldnt DataTable(or DataSet) work better for storing and retreiving this information? Not scrollable, but its searchable.

 

[Joe Mamma]

I dont understand. . .

you talk asciii, but then talk byte data.
is the file delimited/fixed width (ASCII)? or structured:
Double,Byte,Integer,char(2),Byte, Byte(8) - therefore binary?

which ever it is makes a difference as to how to approach it

 

[donnacha]

Can you store the info in the file as xml, then you will be able to load the data straight into a datasat in 1 go and then bind this dataset to whatever controls you want. This would be the most efficent way to do it.

 

[Denaes]

Can you store the info in the file as xml, then you will be able to load the data straight into a datasat in 1 go and then bind this dataset to whatever controls you want. This would be the most efficent way to do it.

More efficient, if you just want the data in a control, would be a datareader, loading each record.

If you want to keep the data in memory, then the dataset is just fine.

still you normally dont want to display 1.5 million records at one time.

 

[michael_hk]

I dont think anyone will like to have so many items in a listbox....

 

[Derek Stone]

XML will never be performant with the exception of binary XML (see W3C XML Binary Characterization Group). Suggesting XML as a solution to a performance problem is usually, if not always, misguided.

 

[donnacha]

One basic question for you, why do you not use a database instead of a text file. It might make life easier for you. Are you restricted byhow the data is put into this file.