• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
  • Campbell Ritchie
  • Paul Clapham
  • Ron McLeod
  • Tim Cooke
  • Junilu Lacar
  • Rob Spoor
  • Devaka Cooray
  • Jeanne Boyarsky
Saloon Keepers:
  • Jesse Silverman
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
  • Tim Holloway
  • Jj Roberts
  • Al Hobbs
  • Piet Souris

Output of reducer sent to HDFS where as map output is stored in data node local disk?

Posts: 2
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi All

I am bit confused about HDFS storage and Data node storage. Below are my doubts.

1. Map function output will be saved to data node local disk and reducer output will be sent to HDFS. As we all know that data blocks are stored in data nodes local disk is there any
other disk space available for HDFS in data node??

2. What is the physical storage location of reducer output file (part-nnnnn-r-00001) ? will it be stored in Name node hard disk?

So my assumption is data node is part is of HDFS i assume data node local disk is also part of HDFS.

Ranch Hand
Posts: 172
Redhat Ruby C++
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I would make a summary that I think it may help you. the Map input OR Reduce output can be from any source (local hist or HDFS, for this example). You configure from/to where it will get/put the input/output. Also. consider the data nodes as a whole (a cluster) , it all is the hdfs. Each node have a piece of your data (or not).
You showed up just in time for the waffles! And this tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
    Bookmark Topic Watch Topic
  • New Topic