Granny's Programming Pearls
"inside of every large program is a small program struggling to get out"
JavaRanch.com/granny.jsp
The moose likes Java in General and the fly likes StringTokenizer Bug? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Java 8 in Action this week in the Java 8 forum!
JavaRanch » Java Forums » Java » Java in General
Bookmark "StringTokenizer Bug?" Watch "StringTokenizer Bug?" New topic
Author

StringTokenizer Bug?

Ryan Bailey
Ranch Hand

Joined: Feb 26, 2002
Posts: 134
Ok simple short and sweet.
I'm writting an application that used the StringTokenizer to parse thru data. I am delimiting it by comma (,).

Example Data:
"a,b,c,,e"
When I iterate thru the String I would expect the StringTokenizer to process this result:
"a" "b" "c" "" "e"
Instead I'm getting:
"a" "b" "c" "e"

Any idea why the StringTokenizer isn't tokenizing empty Strings??? I looked in the javadoc and it says nothing about ignoring empty Strings. I guess I could write my own tokenizer or do something to the data before tokenizing to place N/A where data doesn't exist. But don't want to go to the trouble if it's handled somehow in the StringTokenizer class.

Regards,
Ryan


SCJP 1.4, SCWCD
Java: The power, elegance, and simplicity of a hand grenade
Shailesh Chandra
Ranch Hand

Joined: Aug 13, 2004
Posts: 1081

Originally posted by Ryan Bailey:
Any idea why the StringTokenizer isn't tokenizing empty Strings???

Yes this a limitation of StringTokenizer or bug if you say.I have read somewhere that StringTokenizer is a legacy class and sun discourage use of it, but hasn't deprecated this so far

Originally posted by Ryan Bailey:
I guess I could write my own tokenizer or do something to the data before tokenizing


You need not to write your own tokenizer if you are using jdk1.4 as in JDK 1.4 String class has a split method.
also split method consider empty tokens in a string.

Shailesh
[ April 30, 2005: Message edited by: Shailesh Chandra ]

Gravitation cannot be held responsible for people falling in love ~ Albert Einstein
Ryan Bailey
Ranch Hand

Joined: Feb 26, 2002
Posts: 134
LOL... So much for my SuperStringTokenizer class

Thanks for the reply!
Ryan
David Harkness
Ranch Hand

Joined: Aug 07, 2003
Posts: 1646
By default, StringTokenizer groups one or more consecutive delimiter characters together to separate tokens. However, it has an alternate method that will return both the tokens and each single delimeter character, also as a token.
If the returnDelims flag is true, then the delimiter characters are also returned as tokens. Each delimiter is returned as a string of length one. If the flag is false, the delimiter characters are skipped and only serve as separators between tokens.
While not as simple to use as String.split(","), it would allow you to determine when to create empty tokens. If you have to run in pre-1.4 JVMs, it's a reasonable route. Of course, if comma is your only delimiter character, you could write your own split method in a few lines of code.
[ April 30, 2005: Message edited by: David Harkness ]
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: StringTokenizer Bug?
 
Similar Threads
please help with tokenizing
String splitting & tokenizing problem
Tokenizing
Tokenizing
Tokenizing after reading from a file