Win a copy of Mesos in Action this week in the Cloud/Virtualizaton forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Pig - When will it support UTF-16 ?

 
Abhi murali
Greenhorn
Posts: 12
3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Alan,

Congrats on the release of the book!!!

I currently am working on information retrieval and processing on sanskrit documents and plan to use hadoop in the near future.

I went through the documentation and found that the chararray supports only UTF-8 strings.
When will Pig support UTF-16 characters?
does it help in information retrieval and processing ?

(As of now,I have very limited knowledge about Hadoop..)

Thanks,
Abhinandan
 
Alan Gates
author
Greenhorn
Posts: 7
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You just need to read strings that are UTF-16 encoded? You should be able to write a loader (details are in the book and the online documentation) that assumes strings are UTF-16 encoded. Once they've been converted to a Java String Pig will handle them just fine, even though internally it will move them to UTF-8 when it stores them.

In your second question, "does it help in information retrieval and processing", is "it" Pig or UTF-16 support? I'm guessing Pig. Pig can be used to load data into Hadoop (again via custom load functions) but its main focus is processing data once the data is on Hadoop. Many people use other tools to get their data into Hadoop first, and then process it with Pig.

Alan.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic