File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
The moose likes Distributed Java and the fly likes Pig - When will it support UTF-16 ? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Distributed Java
Bookmark "Pig - When will it support UTF-16 ?" Watch "Pig - When will it support UTF-16 ?" New topic

Pig - When will it support UTF-16 ?

Abhi murali

Joined: Jan 28, 2010
Posts: 12
Hi Alan,

Congrats on the release of the book!!!

I currently am working on information retrieval and processing on sanskrit documents and plan to use hadoop in the near future.

I went through the documentation and found that the chararray supports only UTF-8 strings.
When will Pig support UTF-16 characters?
does it help in information retrieval and processing ?

(As of now,I have very limited knowledge about Hadoop..)

Alan Gates

Joined: Nov 29, 2011
Posts: 7
You just need to read strings that are UTF-16 encoded? You should be able to write a loader (details are in the book and the online documentation) that assumes strings are UTF-16 encoded. Once they've been converted to a Java String Pig will handle them just fine, even though internally it will move them to UTF-8 when it stores them.

In your second question, "does it help in information retrieval and processing", is "it" Pig or UTF-16 support? I'm guessing Pig. Pig can be used to load data into Hadoop (again via custom load functions) but its main focus is processing data once the data is on Hadoop. Many people use other tools to get their data into Hadoop first, and then process it with Pig.

I agree. Here's the link:
subject: Pig - When will it support UTF-16 ?
It's not a secret anymore!