tokenizer

[henry wu]

Hi everyone,
I made a StreamTokenizer object to capture keyboard input from the user. I want to be able to input numbers like "001" and "002" but the in tokenizer.nval it only captures "1" and "2". Is there any way around this?
StreamTokenizer tokenizer = new StreamTokenizer(new InputStreamReader(System.in));
if (tokenizer.nextToken()==tokenizer.TT_NUMBER)
{ return (int)tokenizer.nval; }

[Dan Chisholm]

It sounds like you want a string representation of the input numbers so why not just read them as a String using nextToken()?

[henry wu]

I tried doing this:
if (tokenizer.nextToken()==tokenizer.TT_WORD) {
return tokenizer.sval;
}
else {
System.out.println("This is not a valid string\n");
}
When I entered in '001' It told me that it wasn't a valid string.. so it didn't pass the tokenizer.TT_WORD test. Any ideas?

[Jim Yingst]

Evidently StreamTokenizer considers 001 to be a number, not a word, and it considers 001 to be exactly equivalent to 1. If it's important to you that 001 is different from 1, you'll need to tell the StreamTokenizer to treat numeric characters as parts of words:

    tokenizer.wordChars('0', '9');

[henry wu]

cool thanks.. I'll try this out!

[henry wu]

Ok the new code looks like this:
tokenizer.wordChars('0','9');
if (tokenizer.nextToken()==tokenizer.TT_WORD) {
return tokenizer.sval;
}
else {
System.out.println("This is not a valid string\n");
}
But it is still not recognizing '001' as a word.. Am I placing the wordChars function call in the wrong place?

[Thomas Paul]

wordChars takes two ints as parameters. Try it without the single quotes.

[henry wu]

Ok, changed to -
tokenizer.wordChars(0,9);
if (tokenizer.nextToken()==tokenizer.TT_WORD) {
return tokenizer.sval;
}
else {
System.out.println("This is not a valid string\n");
}
Still didn't work..
Do you guys have any recommendations for keyboard input for integers and strings other than StreamTokenizer and BufferedReader?

[Jim Yingst]

No, Thomas is on the wrong trail here. Passing int values of 0 and 9 just ends up referring to Unicode values of 0-9. You definitely want the char values, which are '0' and '9'. I think they only use int in the API so that people can use int literals without casting if they wish. Rather silly, and misleading.
Looking further in the API, it appears that characters can be both number chars and word chars simultaneously. Evidently the code checks for number chars before it checks for word chars, and so it ignores the fact that we told it that '0'-'9' are word chars. Instead, we must explicitly tell the tokenizer that '0'-'9' are not numbers:

    tokenizer.ordinaryChars('0','9');
    tokenizer.wordChars('0','9');

Note that ordinaryChars() wipes out any special attributes associated with the given chars, so it's necessary to assert the word nature of the chars after ordinaryChars().
On inspection, it appears that StreamTokenizer has one of the more bizarre and confusing API's I've seen from Sun. Maybe even worse than Calendar. Ugh. Depending what you're doing, you may well find it's easier to write your own parsing classes.

[henry wu]

cool.. that worked! Thanks for all the help!
should i have used stringTokenizer from the start?
[ July 10, 2002: Message edited by: henry wu ]