wood burning stoves 2.0*
The moose likes Hadoop and the fly likes Output of  reducer sent to HDFS where as map output is stored in data node local disk? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Murach's Java Servlets and JSP this week in the Servlets forum!
JavaRanch » Java Forums » Databases » Hadoop
Bookmark "Output of  reducer sent to HDFS where as map output is stored in data node local disk?" Watch "Output of  reducer sent to HDFS where as map output is stored in data node local disk?" New topic
Author

Output of reducer sent to HDFS where as map output is stored in data node local disk?

Sureshdv Babu
Greenhorn

Joined: Sep 23, 2011
Posts: 2
Hi All

I am bit confused about HDFS storage and Data node storage. Below are my doubts.

1. Map function output will be saved to data node local disk and reducer output will be sent to HDFS. As we all know that data blocks are stored in data nodes local disk is there any
other disk space available for HDFS in data node??

2. What is the physical storage location of reducer output file (part-nnnnn-r-00001) ? will it be stored in Name node hard disk?


So my assumption is data node is part is of HDFS i assume data node local disk is also part of HDFS.
Regards
Suresh


Luan Cestari
Ranch Hand

Joined: Feb 07, 2010
Posts: 162

I would make a summary that I think it may help you. the Map input OR Reduce output can be from any source (local hist or HDFS, for this example). You configure from/to where it will get/put the input/output. Also. consider the data nodes as a whole (a cluster) , it all is the hdfs. Each node have a piece of your data (or not).


Please, visit me for some cool tech post at www.ourdailycodes.com
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Output of reducer sent to HDFS where as map output is stored in data node local disk?
 
Similar Threads
Retrieving map reduce results
mapReduce job output
Using Hadoop to process large text files along with CSV
Big Data Real-time Application on Database
How to save the output of Reducer of a Hadoop Map Reduce as csv