• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Paul Clapham
  • Ron McLeod
  • Jeanne Boyarsky
  • Tim Cooke
Sheriffs:
  • Liutauras Vilda
  • paul wheaton
  • Henry Wong
Saloon Keepers:
  • Tim Moores
  • Tim Holloway
  • Stephan van Hulst
  • Carey Brown
  • Frits Walraven
Bartenders:
  • Piet Souris
  • Himai Minh

StringTokenizer - Doesn't return empty Strings or nulls

 
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Everyone,
I'm currently teaching my colleagues how to do Java, and yesterday I was pointing out some examples in their code where they could have used classes that already exist, rather than reinventing the wheel. In particular, I pointed out a passage of code where someone had rewritten the StringTokenizer. I launched into a description of the reasons why using the StringTokenizer would be better, but was stopped in my tracks. Someone pointed out that, where the String passed in contains two delimiters next to each other, the StringTokenizer returns nothing at all, which was not the desired behaviour. I hadn't ever encountered this problem, and assumed there must be a flag you can set to make it return an empty String or null in this situation. However, I checked the API docs (1.4) and, lo and behold, no such flag exists. I can't believe this problem has never been encountered before. Does anyone know whether it has been addressed anywhere in core Java or elsewhere? If you do, please tell me, so I can tell my colleagues and not look such an idiot.
Thanks!
 
Ranch Hand
Posts: 75
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
//where the String passed in contains two delimiters next to each other, the StringTokenizer returns nothing at all, which was not the desired behaviour
The StringTokenizer is a very simple class. Used normally a token is a "maximal sequence of consecutive characters that are not delimiters" so if there is no space between the delimiter then there is no token. Set the parameter returnDelims to true, and the delimiters themselves are returned as tokens. For example:
String test = "a,b,c,,d";
StringTokenizer defaultBehaviour = new StringTokenizer(test, ",");
while (defaultBehaviour.hasMoreTokens()) {
System.out.println(defaultBehaviour.nextElement());
}
// outputs :
// a
// b
// c
// d
StringTokenizer withDelimsBehaviour = new StringTokenizer(test, ",", true);
while (withDelimsBehaviour.hasMoreTokens()) {
System.out.println(withDelimsBehaviour .nextElement());
}
// outputs:
// a
// ,
// b
// ,
// c
// ,
// ,
// d
So you'll be able to catch empty elements with the second tokenizer. If you need more advanced tokenizing functionality use a StreamTokenizer.
 
Ranch Hand
Posts: 331
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I had the same frustration when I first discovered that "feature" of StringTokenizer. I haven't used it yet, but I believe the split() method of String in 1.4 doesn't do that, i.e. empty tokens are returned.
 
Ranch Hand
Posts: 399
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I think we've all run into the "special" feature of StringTokenizer and cursed Sun. You can do as John indicates, but then you have to put in lots of special code to decide, "is this token a delimiter or a value that I care about."
If you can use JDK1.4, do as Blake suggests and use the String.split() method. There are a couple of things to keep in mind. If you pass "xy" as the delimiter to StringTokenizer, it means that use either "x" OR "y" as a delimiter. If you pass it to split, it means use "xy" as the delimiter. So the semantics are different.
Even with "split()", the default behavior won't give you TRAILING null tokens. In other words, if you want to split the string "a:b::c::" based on a colon, you'll get "a", "b", "", and "c", but not the two trailing ones. You have to pass in a negative one (-1) as the optional second parameter to get those last two empty tokens.
Also, the pattern you pass in for the "split()" method is treated as a regular expression, so if you want to split it at a pipe (|) you have to use "\\|" as the pattern. Yes, both back-slashes are required.
[ March 09, 2004: Message edited by: Wayne L Johnson ]
 
Ranch Hand
Posts: 1179
Mac OS X Eclipse IDE
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Try download this util API Utilities

The result will then be:
a
b
c
d
Ren´┐Ż
 
Roses are red, violets are blue. Some poems rhyme and some don't. And some poems are a tiny ad.
free, earth-friendly heat - a kickstarter for putting coin in your pocket while saving the earth
https://coderanch.com/t/751654/free-earth-friendly-heat-kickstarter
reply
    Bookmark Topic Watch Topic
  • New Topic