Difference between split() and StringTokenizer s nextToken()

[Deepak Kumar]

Hi All,

I have the following program,

The string "x" is a tab separated
1<tab>2<tab>s<tab><tab>3<tab><tab><tab>4<tab>;

 public class Testoken {

    public static void main(String[] args) {
        String x = "1	2	s		3			4	";
        StringTokenizer st = new StringTokenizer(x,"\t");
        int i = 0;
        while(st.hasMoreTokens()){
            System.out.println("token-->"+st.nextToken());            
            i++;
        }
        System.out.println("i-->"+i);//elements from tokenizer
        String [] a = x.split("\t");
        System.out.println("length--->"+a.length);
        for(int y = 0;y<a.length;y++){
        System.out.println("value-->"+a[y]);//elements from split
        }
    }
}   

Output:

token-->1
token-->2
token-->s
token-->3
token-->4
i-->5
length--->8
value-->1
value-->2
value-->s
value-->
value-->3
value-->
value-->
value-->4



Why there is a discrepency in the output from tokenizer and split?

Thanks in advance.

Regards,
Deepak

[Rob Spoor]

String.split() does it the right way. Between the s and 3, there are two tabs, and between the 3 and 4 there are three tabs. Between those tabs, there are actually empty strings. Those show up, whereas StringTokenizer simply ignores them.

Now, if you want to remove those empty strings, change the regular expression of String.split to "\t+". The + means once or more.

[Deepak Kumar]

Hi Rob,

Since split is returning 8 elements and tokenizer is returning 5 element.
So How can I return 8 elements with use of StringTokenizer only?

Regards,
Deepak

[Rob Spoor]

One way would be to use the constructor with a boolean, and make the StringTokenizer return the delimiters as well. But then you'd have to handle the delimiters as special cases. Not a nice solution at all.

The other option (and my favourite): don't use StringTokenizer at all. String.split does just what you want, and you can mimick StringTokenizer's behaviour very easily with String.split; the other way around is harder as you have just noticed.