StringTokenizer

[Shane Roylance]

Hi all,
Hopefully this is a really simple question for someone out there. I have a string that I am parsing that looks like this "some string<token><token>Another string<token>Last string<token>" or at times it could look like this. The place where two tokens are together is where I have a blank string. Using the nextToken() method of StringTokenizer, it seems to skip the blank string, so the second string I get from next token is "Another string" . Am I halucinating, or is this how it is supposed to work. It seems to me that if two tokens are next to each other it would return a blank string.
Anyone know how this is really supposed to work?
Thanks
Shane

[Dirk Schreckmann]

Note that StringTokenizer has a constructor that takes a boolean as the last argument. It might provide the function that you are after.
Otherwise, use of the java.util.regex package might be more appropriate to solve your problem.

import java.util.regex.Pattern ;
 
public class Foo 
{
    public static void main( String[] args ) 
    {
        String input = "some string<token><token>Another string<token>Last string<token>" ;
        
        Pattern pattern = Pattern.compile( "<token>" );
        String[] splits = pattern.split( input );
        
        for ( int i = 0 , length = splits.length ; i < length ; i++ ) 
        {
            System.out.println( splits[ i ] );
        }
    }
}

Displays:
some string
Another string
Last string
[ August 19, 2002: Message edited by: Dirk Schreckmann ]

[Jim Yingst]

[I swear, Dirk's code sample wasn't there when I first wrote this...]
StringTokenizer is pretty inconvenient for this sort of thing. That boolean argument Dirk mentions modifies its behavior, but it's still not very easy to use. There are various workarounds, but probably the easiest is to use the String split() method in JDK 1.4:

    String[] fields = inputStr.split("<token>");
    for (int i = 0; i < fields.length; i++)
    {
      process(fields[i]);
    }

This has two big advantages: (1) it handles multi-character tokens the way most people would expect, and (2) if two tokens are next to one another it returns a blank string for that field.
[ August 19, 2002: Message edited by: Jim Yingst ]

[Shane Roylance]

Thank you both,
Two solutions that both work great.
Thanks again.