Stream Tokenizer - Special case

Hi,
In my application we are using StreamTokenizer to
read a file. We are using the constructor
StreamTokenizer(Reader r).
The file is in comma separated format. OUr requirement is to split the data into tokens using comma(,) as the whitespace character. All other characters are made as ordinary characters
using the method resetSyntax(). Then we use the next Token method in a loop to extract the data.
But the problem arises when the data in the file
is like this:
40,,44,56,,,78,,,,
Note that there is nothing between the commas.
In this case the nextToken method just skips these empty data cells. But we need to identify these empty or null data. Please help with any suggestions. I am not sure if StreamTokenizer is going to help.
The code :->
public class MyParser {
private StreamTokenizer parser;

public MyParser() {
super();
}
public MyParser(Reader in) {
super();
if (!(in instanceof BufferedReader)) {
BufferedReader br = new BufferedReader(in);
parser = new StreamTokenizer(br);
}
else parser = new StreamTokenizer(in);
init();
}

public String[] getRow() throws IOException {
List results = new ArrayList();
if ((parser.ttype == 0) || (parser.ttype == parser.TT_EOL)) parser.nextToken();

while ((parser.ttype != parser.TT_EOL) && (parser.ttype != parser.TT_EOF)){

if (parser.sval != null) results.add(parser.sval);
parser.nextToken();

}
String[] retArray = new String[results.size()];
for (int i = 0; i < results.size(); i++){
retArray[i] = (String) results.get(i);
}
return (retArray);
}
public boolean hasMoreRows(){
return !(parser.ttype == parser.TT_EOF);
}
private void init() {

parser.resetSyntax();
parser.whitespaceChars(',', ',');
parser.quoteChar('"');
parser.wordChars(' ',' ');
parser.wordChars(32,43);
parser.wordChars(45,255);
parser.eolIsSignificant(true);

}
public static void main(String[] args) {
try {

URL url = new URL("ftp//myserver");
url.openConnection();
InputStream ftp = url.openStream();
InputStreamReader ftpr = new InputStreamReader(ftp);
MyParser my = new MyParser(ftpr);
java.io.LineNumberReader ln = new java.io.LineNumberReader(ftpr);

do {

String[] st = my.getRow();
System.out.println("START NEW ROW"+ " " + st.length);
for (int i =0; i < st.length; i++){
System.out.println(st[i] + " " + st[i].length());

}

} while (my.hasMoreRows());
}
catch (Throwable t){
t.printStackTrace();
}

}empty or null

What about something simple...
Before parsing the text with StreamTokenizer, go through it and add a space (or anything you'd like) between two consecutive commas.

Hi Dirk,
How do I go about doing this. Can you give me an example. How can I parse through the stuff. Where can I store the output stream after I parse it. I cannot store it on the disk and read back from it to parse it again for a second time.
I appreciate your help
Thanks
Sri

Originally posted by Dirk Schreckmann:
What about something simple...
Before parsing the text with StreamTokenizer, go through it and add a space (or anything you'd like) between two consecutive commas.

Does any one know if there is Open Souce code for CSV parsers.
Your help is very much appreciated.
Thanks
Sri

Hi,
I gave up using StreamTokenizer as it's so quirky.
Couldn't you use StringTokenizer instead, then you can write your own methods to handle your specific data coming through.
Hope this helps.
Terence

You could use a StringBuffer to hold your text data while you are manipulating it. Here's a simple for loop that would do the insertion of a space character between any two consecutive commas:

Now, I have a philosophy paper due, will you write it for me?

I wrote an open source comma separated value parser:
http://ostermiller.org/utils/CSVLexer.html