File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Beginning Java and the fly likes tokenizer Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Beginning Java
Bookmark "tokenizer" Watch "tokenizer" New topic
Author

tokenizer

henry wu
Greenhorn

Joined: Dec 01, 2000
Posts: 23
Hi everyone,
I made a StreamTokenizer object to capture keyboard input from the user. I want to be able to input numbers like "001" and "002" but the in tokenizer.nval it only captures "1" and "2". Is there any way around this?
StreamTokenizer tokenizer = new StreamTokenizer(new InputStreamReader(System.in));
if (tokenizer.nextToken()==tokenizer.TT_NUMBER)
{ return (int)tokenizer.nval; }
Dan Chisholm
Ranch Hand

Joined: Jul 02, 2002
Posts: 1865
It sounds like you want a string representation of the input numbers so why not just read them as a String using nextToken()?


Dan Chisholm<br />SCJP 1.4<br /> <br /><a href="http://www.danchisholm.net/" target="_blank" rel="nofollow">Try my mock exam.</a>
henry wu
Greenhorn

Joined: Dec 01, 2000
Posts: 23
I tried doing this:
if (tokenizer.nextToken()==tokenizer.TT_WORD) {
return tokenizer.sval;
}
else {
System.out.println("This is not a valid string\n");
}
When I entered in '001' It told me that it wasn't a valid string.. so it didn't pass the tokenizer.TT_WORD test. Any ideas?
Jim Yingst
Wanderer
Sheriff

Joined: Jan 30, 2000
Posts: 18671
Evidently StreamTokenizer considers 001 to be a number, not a word, and it considers 001 to be exactly equivalent to 1. If it's important to you that 001 is different from 1, you'll need to tell the StreamTokenizer to treat numeric characters as parts of words:


"I'm not back." - Bill Harding, Twister
henry wu
Greenhorn

Joined: Dec 01, 2000
Posts: 23
cool thanks.. I'll try this out!
henry wu
Greenhorn

Joined: Dec 01, 2000
Posts: 23
Ok the new code looks like this:
tokenizer.wordChars('0','9');
if (tokenizer.nextToken()==tokenizer.TT_WORD) {
return tokenizer.sval;
}
else {
System.out.println("This is not a valid string\n");
}
But it is still not recognizing '001' as a word.. Am I placing the wordChars function call in the wrong place?
Thomas Paul
mister krabs
Ranch Hand

Joined: May 05, 2000
Posts: 13974
wordChars takes two ints as parameters. Try it without the single quotes.


Associate Instructor - Hofstra University
Amazon Top 750 reviewer - Blog - Unresolved References - Book Review Blog
henry wu
Greenhorn

Joined: Dec 01, 2000
Posts: 23
Ok, changed to -
tokenizer.wordChars(0,9);
if (tokenizer.nextToken()==tokenizer.TT_WORD) {
return tokenizer.sval;
}
else {
System.out.println("This is not a valid string\n");
}
Still didn't work..
Do you guys have any recommendations for keyboard input for integers and strings other than StreamTokenizer and BufferedReader?
Jim Yingst
Wanderer
Sheriff

Joined: Jan 30, 2000
Posts: 18671
No, Thomas is on the wrong trail here. Passing int values of 0 and 9 just ends up referring to Unicode values of 0-9. You definitely want the char values, which are '0' and '9'. I think they only use int in the API so that people can use int literals without casting if they wish. Rather silly, and misleading.
Looking further in the API, it appears that characters can be both number chars and word chars simultaneously. Evidently the code checks for number chars before it checks for word chars, and so it ignores the fact that we told it that '0'-'9' are word chars. Instead, we must explicitly tell the tokenizer that '0'-'9' are not numbers:

Note that ordinaryChars() wipes out any special attributes associated with the given chars, so it's necessary to assert the word nature of the chars after ordinaryChars().
On inspection, it appears that StreamTokenizer has one of the more bizarre and confusing API's I've seen from Sun. Maybe even worse than Calendar. Ugh. Depending what you're doing, you may well find it's easier to write your own parsing classes.
henry wu
Greenhorn

Joined: Dec 01, 2000
Posts: 23
cool.. that worked! Thanks for all the help!
should i have used stringTokenizer from the start?
[ July 10, 2002: Message edited by: henry wu ]
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: tokenizer