Suggestions on fastest way to parse a String?

[Ron Ditch]

Hello...
Does anyone have any suggestions on the fastest (and hopefully most efficient) way to parse a string?
Let's say I have a string that is comma delimited, and I wanted to convert it to a Collection. Also, the elements in the string that are comma delimited are of unequal length.
For example - item1,items22,item333,item55555
I was thinking of using an array of characters, but I don't know the speed implication of for loops versus creating sub-strings using String.substring(int,int).
Any suggestions?

[Ilja Preuss]

Use java.util.StringTokenizer - it's optimized for exactly this type of parsing.
[ September 26, 2002: Message edited by: Ilja Preuss ]

[Blake Minghelli]

Just a warning about StringTokenizer if you have never used it before...
The default behavior ignores empty "tokens".
For example: "token1,token2,,token3"
A StringTokenizer created on that string will return 3 tokens.

[Jim Yingst]

If you really want the fastest parsing possible, you can probably improve on StringTokenizer a little bit, because StringTokenizer spends a little bit of time checking for multiple delimiters, and even checking to see if the set of delimiters has changed since the last time nextToken() was called. You can omit this for your situation, and thereby speed things up a bit, I imagine. But I doubt you'll see a big difference, so don't spend too much time on it unless you're sure performance is a real problem. I'd probably just store the input as a String, and use indexOf(',', startPos) to find delimiters, and substring(int, int) to create a String for each token. You could also store the input as a char[] array; I'm not sure if that will end up any faster or not. You'd have to try both ways and measure, I suppose.
Now in terms of development speed (rather than execution speed), the easiest solution is probably
String[] tokens = inputStr.split(",");
Try it; you may well find it's already fast enough for you. (You need to be using SDK 1.4 though.) It also fixes the annoying "feature" of StringTokenizer which Blake mentioned.

[Ron Ditch]

Thanks Jim, that's what I was looking for.

[Thomas Paul]

You should keep in mind that StringTokenizer was designed to parse Java programs. The token to split on was assumed to be a space. The reason we have the default behavior of the StringTokenizer is that multiple spaces doesn't mean anything special in java source.

[Yarik Chinskiy]

Hi,
What if i want to parse records of a file?
whouldn't the StringTokenizer be a killer??
I want to monitor a log file and reformat the records for the output based on a pattern submitted by a user.

[Jim Yingst]

Tom's comment may be a bit misleading - it's possible to use StringTokenizer to parse a lot of things other than Java code. But it has a number of limitations - nowadays it's probably more powerful and flexible to learn how to parse using the classes in java.util.regex (at least, for anything more complicated than the split() method I showed above).

[John Coffey]

I have some sample code to test out "log" parsing. It looks like StringTokenizer isn't too good as far as performance is concerned. Using jdk 1.4.1, I got the following results:

Split           = 24175 ms
StringTokenizer = 13920 ms
Pos             = 10235 ms

Can anyone come up with a faster version? Is there a better IO class?
First a utility to create a big log file:

import java.io.*;
import java.util.*;
public class MakeBig {
   public static void main(String[] args) throws IOException {
      String s;
      String sev[] = {"INFO","WARNING","ERROR","FATAL"};
      Date d;
      int r;
      int i=0;
      while (i<1000000) {
         d = new Date();
         r = (int)(Math.random()*4);
         i = i+1;
         System.out.println(d+","+i+","+sev[r]+",This is a test message");
      }
   }
}

Now the Split code:

import java.io.*;
public class Split {
   public static void main(String args[]) throws Exception {
      String s;
      String[] tokens;
      int count = 0;
      long t1 = System.currentTimeMillis();
      BufferedReader in = new BufferedReader(
         new FileReader("big.log"));
      while ((s = in.readLine()) != null) {
         tokens = s.split(",");
         if (tokens[2].equals("FATAL")) count++;
      }
      System.out.println("count = "+count);
      long t2 = System.currentTimeMillis();
      long delta = t2 - t1;
      System.out.println("Split = "+delta);
   }
}

Now the StringTokenizer code:

 
import java.io.*;
import java.util.*;
public class Tokenizer {
   public static void main(String args[]) throws Exception {
      String s;
      StringTokenizer tok;
      String s0, s1, s2, s3;
      int count = 0;
      long t1 = System.currentTimeMillis();
      BufferedReader in = new BufferedReader(
         new FileReader("big.log"));
      while ((s = in.readLine()) != null) {
         tok = new StringTokenizer(s,",");
         s0 = tok.nextToken();
         s1 = tok.nextToken();
         s2 = tok.nextToken();
         s3 = tok.nextToken();
         if (s2.equals("FATAL")) count++;
      }
      System.out.println("count = "+count);
      long t2 = System.currentTimeMillis();
      long delta = t2 - t1;
      System.out.println("Tokenizer = "+delta);
   }
}

Now the Pos code:

import java.io.*;
public class Pos {
   public static void main(String args[]) throws Exception {
      String s;
      int pos1, pos2, pos3, pos4;
      String s0, s1, s2, s3;
      int count = 0;
      long t1 = System.currentTimeMillis();
      BufferedReader in = new BufferedReader(
         new FileReader("big.log"));
      while ((s = in.readLine()) != null) {
         pos1 = s.indexOf(',');
         pos2 = s.indexOf(',',pos1+1);
         pos3 = s.indexOf(',',pos2+1);
         pos4 = s.indexOf(',',pos3+1);
         s0 = s.substring(0, pos1);
         s1 = s.substring(pos1+1, pos2);
         s2 = s.substring(pos2+1, pos3);
         s3 = s.substring(pos3+1);
         if (s2.equals("FATAL")) count++;
      }
      System.out.println("count = "+count);
      long t2 = System.currentTimeMillis();
      long delta = t2 - t1;
      System.out.println("Pos = "+delta);
   }
}