File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
The moose likes Java in General and the fly likes Issue with StreamTokenizer? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Java in General
Bookmark "Issue with StreamTokenizer?" Watch "Issue with StreamTokenizer?" New topic

Issue with StreamTokenizer?

Santhosh Kumar
Ranch Hand

Joined: Nov 07, 2000
Posts: 242
I was fooling around with StreamTokenizer to create a SubClass, which can return comments as tokens. In the process, I stumbled into something very puzzling. See the code.

Executing the above code displays:

While parsing the "hello" and "hello, why didn't it return the hello world token?

Any idea?

[ June 28, 2008: Message edited by: Santhosh Kumar ]
[ June 28, 2008: Message edited by: Santhosh Kumar ]
Alan Moore
Ranch Hand

Joined: May 06, 2004
Posts: 262
In the case of "hello", it's because the token is a quoted string, not a word. The API doc of the ttype field says
For a quoted string token, its value is the quote character.
As for "hello, I'd say the output is meaningless since the input is malformed (I'd say the same about hello", too).
Bill Shirley
Ranch Hand

Joined: Nov 08, 2007
Posts: 457
To restate Alan's reply, the documentation for the ttype instance variable gives the information.

This value is also what is being returned and you are setting to token.

There are two values that the tokenizer doesn't define a special code for. The character is your fall-through catch, but the other is the quoted string (which they should have defined another value for - i did it in my modification to your code...).

Here's the code:

Bill Shirley - bshirley -
if (Posts < 30) JavaRanchFAQ);
Santhosh Kumar
Ranch Hand

Joined: Nov 07, 2000
Posts: 242
Thank you. That was really helpful.
I agree. Here's the link:
subject: Issue with StreamTokenizer?
It's not a secret anymore!