File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Issue with StreamTokenizer?

 
Santhosh Kumar
Ranch Hand
Posts: 242
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I was fooling around with StreamTokenizer to create a SubClass, which can return comments as tokens. In the process, I stumbled into something very puzzling. See the code.



Executing the above code displays:



While parsing the "hello" and "hello, why didn't it return the hello world token?

Any idea?

[ June 28, 2008: Message edited by: Santhosh Kumar ]
[ June 28, 2008: Message edited by: Santhosh Kumar ]
 
Alan Moore
Ranch Hand
Posts: 262
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
In the case of "hello", it's because the token is a quoted string, not a word. The API doc of the ttype field says
For a quoted string token, its value is the quote character.
As for "hello, I'd say the output is meaningless since the input is malformed (I'd say the same about hello", too).
 
Bill Shirley
Ranch Hand
Posts: 457
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
To restate Alan's reply, the documentation for the ttype instance variable gives the information.

This value is also what is being returned and you are setting to token.

There are two values that the tokenizer doesn't define a special code for. The character is your fall-through catch, but the other is the quoted string (which they should have defined another value for - i did it in my modification to your code...).

Here's the code:
 
Santhosh Kumar
Ranch Hand
Posts: 242
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you. That was really helpful.
 
I agree. Here's the link: http://aspose.com/file-tools
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic