String Tokenizer

[lphung]

Hi folks,
During the use of the StringTokenizer class, I came across a string in the format "1","ABC","Last, First","CAT","MBA"
If i use the comma as a my delimiter, how do I handle the "," in inside the double quoted string? Especially when I do a countTokens(), it returns 6 tokens instead of 5.
Any help would be greatly appreciated.

[Frank Carver]

"Iphung",
The Java Ranch has thousands of visitors every week, many with surprisingly similar names. To avoid confusion we have a naming convention, described at http://www.javaranch.com/name.jsp . We require names to have at least two words, separated by a space, and strongly recommend that you use your full real name. Please log in with a new name which meets the requirements.
Thanks.

[John Bateman]

Hi
I Since everything is a string in your example..
use "," as the 'token'.. BUT, you will have to use indexOf to get a position of each token and then subString() the token out of the String. This is because the StringTokenizer will consider the "," as a list of tokens to check for.
Ex:
"Hi","there","me,you"
Would give you 3 tokens.
token 1> "Hi
token 2> there
token 3> me, you"
Now all you have to do is chop the 1st char off the 1st token and the last char off the last token.
Just my $0.02cdn

[Art Metzer]

Here's all I could think of, and I of course defer to any better solutions.
1. Go through your original String, replacing all commas between quotes with a placeholder character that you can guarantee will never ever appear in any input String.
2. Tokenize this altered String.
3. When you receive each token, replace each placeholder you find with a comma.
4. Process the token normally.
Here's a bit of code to demonstrate:

<font size=2.5>
import java.util.*;
public class CommaTokenizer {
    // Your input String
    public static final String INPUT =
        "\"1\",\"ABC\",\"Last, First\",\"CAT\",\"MBA\"";
    /* PLACEHOLDER must be a char that you can *guarantee*
     * will not ever appear in your input String.  You may want
     * to investigate non-printable char values; I'm using the
     * squiggle because it's easy.  But seriously consider this.
     */
    public static final char PLACEHOLDER = '~';
    public static final char COMMA = ',';
    public static final char QUOTE = '\"';
    /**
     *  @param  The raw String, with commas between quotes
     *  @return The same String with these inter-quote
     *          commas replaced by the char in PLACEHOLDER
     */
    public String deComma( String raw ) {
        char[] withCommas = raw.toCharArray();
        char[] temp = new char[ withCommas.length ];
        boolean withinQuotes = false;
        for ( int i = 0; i < withCommas.length; ++i ) {
            if ( withCommas[i] == QUOTE ) {
                withinQuotes = !withinQuotes;
            }
            if ( ( withCommas[i] == COMMA ) && ( withinQuotes ) ) {
                temp[i] = PLACEHOLDER;
            }
            else {
                temp[i] = withCommas[i];
            }
        }
        return new String( temp );
    }
    /*
     * With this method, do whatever you need to do with your
     * tokens, e.g., strip them of their quotation marks,
     * store in a database, etc.  For simplicity, I just print
     * them to the screen.
     */
    public void process( String fodder ) {
        System.out.println( fodder );
    }
    public static void main( String[] args ) {
        CommaTokenizer ct = new CommaTokenizer();
        // This constructor is a little messy.  The delimiter in
        // StringTokenizer has to be a String, but our COMMA is
        // a char, so for the constructor I have to do this
        // unpleasant conversion.  But at least it only has to
        // be done once.
        StringTokenizer st = new StringTokenizer( ct.deComma( INPUT ),
                                                  new Character( COMMA ).toString() );
        while ( st.hasMoreTokens() ) {
            ct.process( st.nextToken().replace( PLACEHOLDER, COMMA ) );
        }
    }
}

Will something like this do the trick?
Art
[This message has been edited by Art Metzer (edited April 06, 2001).]

[Angela Jessi]

I think you can use indexOf. Then get the length of string.
Here is an example:
int iLen;
iEnd = sToken.indexOf(':', iEnd);
sTemp = sToken.substring(iBegin, iEnd);
iBegin = iEnd + 1;
iEnd++;
if( iFlag == 1 ) // Parsing to get the values of arguments
{
iLen = Integer.parseInt(sTemp);
sTemp = sToken.substring(iBegin, iBegin + Len);
iEnd = iBegin + iLen + 1;
iBegin = iEnd;
}
return sTemp;

Thanks,
Angela