This week's book giveaway is in the Jobs Discussion forum.
We're giving away four copies of Customer Requirements for Developers and have Marcho Behler on-line!
See this thread for details.
The moose likes Other JSE/JEE APIs and the fly likes StringTokenizer and null fields Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login

Win a copy of Customer Requirements for Developers this week in the Jobs Discussion forum!
JavaRanch » Java Forums » Java » Other JSE/JEE APIs
Bookmark "StringTokenizer and null fields" Watch "StringTokenizer and null fields" New topic

StringTokenizer and null fields

Leslie Chaim
Ranch Hand

Joined: May 22, 2002
Posts: 336
I want to parse some CSV text using StringTokenizer, and I am having problem with the null fields. I searched around and found that the StringTokenizer indeed does skip over null fields.
Does anyone know of any API alternative, or must I roll my own little parser as suggested elsewhere.
Your replies are greatly appreciated.

Normal is in the eye of the beholder
Jim Yingst

Joined: Jan 30, 2000
Posts: 18671
If using SDK 1.4, you can use split(String) method in String (and in the java.util.regex classes).

Otherwise rolling your own is best. Or you can use this, which I made some time ago:

[ July 09, 2002: Message edited by: Jim Yingst ]

"I'm not back." - Bill Harding, Twister
William Brogden
Author and all-around good cowpoke

Joined: Mar 22, 2000
Posts: 12983
If I understand your problem correctly, you can use the constructor that takes a returnDelims flag.
public StringTokenizer(String str,
String delim,
boolean returnDelims)Constructs a string tokenizer for the specified string. All characters in the delim argument are the delimiters for separating tokens.
If the returnDelims flag is true, then the delimiter characters are also returned as tokens. Each delimiter is returned as a string of length one. If the flag is false, the delimiter characters are skipped and only serve as separators between tokens.
Then count the delimiters to decide which field you are in.
Jim Yingst

Joined: Jan 30, 2000
Posts: 18671
But the problem with using the returnDelims parameter is that you have to put in extra logic to determine which tokens were delimiters and which weren't. It's not enough to just look at the length, since there might be a non-delimeter field with length 1. And then you still have to put in the logic for actually using the field count. I found it was easier to make my own tokenizer from scratch, rather than adapt the existing StringTokenizer. YMMV of course.
[ July 09, 2002: Message edited by: Jim Yingst ]
Leslie Chaim
Ranch Hand

Joined: May 22, 2002
Posts: 336
We use the 1.3 kit, I hope we upgrade to 1.4 for obvious reasons but the biggest reason would be the regex support.
I actually tried using the constructor with the return delim set to true but as Jim points out there is the extra logic where you almost wonder if the StringTokenizer class is a proper solution.
In my case, since I only have a single character as a delimiter I totally abandoned the StringTokenzier thingJ Instead I used methods of the string class as follows:
String data[] = new String[num_expected_tokens];
int begin_index = 0;
int end_index = line.indexOf('~', begin_index);
for ( int i = 0; i < num_expected_tokens; i++ )
data[i] = line.substring(begin_index, end_index);
begin_index = end_index + 1;
end_index = line.indexOf('~', begin_index);
Yes, in my case I figured out num_expected_tokens elsewhere but I think this solution is much cleaner then using the StringTokenizer class.
Thanks for your help.
Dilanga Thalpegama

Joined: Jul 24, 2013
Posts: 1
I agree. Here's the link:
subject: StringTokenizer and null fields
It's not a secret anymore!