String Tokenizer

[sreeni san]

Hi
I need to tokenize a string with ',' as delimiter.
for example in the string below:
Sreeni, san, 12345, , #44 XYZ street,,234567
Here I need to tokenize with comma as delimiter.
The problem is with the value is not specified and instead when ,, is specified. The StringTokenizer skips this as a value, whereas , , is returned as blank.
How do I overcome this problem, when ,, is specified.
Note: comma character in the above string is not a value.
Thanks
Sreeni

[aminur rashid]

Hi sreeni.
I have tried to solve ur problem.You can have other logic for this also.What i have done in this function is that i have used a temproryTokenizer.From this temproryTokeniser i have eliminated the tokens with values" " and null
Then i have made the finalToknizer

private  void addTokens(String tokenString)
{
StringTokenizer tempTokeniser = new StringTokenizer(tokenString,",");
  String modifiedString =null;
  //counterToChekFirstToken is to chek if the token to be added 
  // to modifiedString is the first token or  not
  int counterToChekFirstToken = 0;
  while(tempTokeniser.hasMoreTokens())
{

 String temp = tempTokeniser.nextToken();
  if(!(temp.trim().equals("")||temp==null))
  {
  if(counterToChekFirstToken==0)
  modifiedString = temp;
  else
  modifiedString = modifiedString+","+temp;
  }
  counterToChekFirstToken++;
}
//Make here retrieves the tokens from the vector in to a String
StringTokenizer finalTokeniser = new StringTokenizer(modifiedString,",");
}

cheers

[sreeni san]

HI
Thanks for the reply.
The problem I have is that I need to retrieve token when the value is ,, . The StringTokenizer just skips this.
I'll try to modify my example:
I have data pertaining to a user for the fields below:
fName, lname, userId, address1, address2,city, phone
The corresponding values for the above fields are
Sreeni, san, 12345, , #44 XYZ street,,234567
Now I tokenize the entire String and assign values to respective variables. While doing so I have encountered problems like for example:
The value for fName, lname, userId, address1, address2 are returned correctly as tokens.
The problem here is with city as the value ",," is skipped and 234567 is returned as next token immediately after address2.
In case of address1 ", ," is value is returned as blank, which is fine for me.
Kindly suggest me a suitable solution.

thanks
Sreeni

[Marilyn de Queiroz]

What if you replace ,, with , , or ,"", or ,null, before you use the StringTokenizer?

[aminur rashid]

Hello the problem with that is ''while ' ' makes a string with a space character.
You can try one thing .Use the constructer

public StringTokenizer(String�str,
                       String�delim,
                       boolean�returnDelims)

It will return values the delimiter
, after every string.Write a method to create a null or "" string if u get two , without getting any values.Hope this will help

[Jim Yingst]

If you're using SDK 1.4, you can handle this easily using the String class' split() method. Otherwise, here's a class I wrote some time ago to solve this problem:

/**
 *  This is a replacement for the standard StringTokenizer which differs
 *  in that if there are multiple delimiters encountered in succession,
 *  nextToken() returns a blank string ("") for each delimiter pair
 *  encountered,  rather than gouping the delimiters together as one,
 *  which is what StringTokenizer does.  Thus the string "A,B,,,E"
 *  breaks into tokens "A", "B", "", "", "E", rather than "A", "B", "E".
 */
public class StrTokenizer {
 
    private String str;
    private String[] delims;
    private int currentPosition;
 
    public StrTokenizer(String str) {
        this(str, " \t\n\r\f");
    }
 
    public StrTokenizer(String str, String delimStr) {
        this.str = str;
        // this.delimStr = delimStr;
        delims = new String[delimStr.length()];
        for (int i = 0; i < delimStr.length(); i++) {
            delims[i] = delimStr.substring(i, i + 1);
        }
    }
 
    public String nextToken() {
        int nextDelimPosition = str.length();
        for (int i = 0; i < delims.length; i++) {
            int delimPosition = str.indexOf(delims[i], currentPosition);
            if (delimPosition >= 0 && delimPosition < nextDelimPosition) {
                nextDelimPosition = delimPosition;
            }
        }
        String token = str.substring(currentPosition, nextDelimPosition);
        currentPosition = nextDelimPosition + 1;
        return token;
    }
 
    public boolean hasMoreTokens() {
        return (currentPosition < str.length());
    }
}

[ April 03, 2002: Message edited by: Jim Yingst ]

[Steve Deadsea]

I wrote a stringtokenizer that can handle null tokens:
http://ostermiller.org/utils/StringTokenizer.html
I also wrote a CSV parser that will even handle quoted values:
http://ostermiller.org/utils/CSVLexer.html