File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Hadoop and the fly likes Decommission of Node Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of The Java EE 7 Tutorial Volume 1 or Volume 2 this week in the Java EE forum
or jQuery UI in Action in the JavaScript forum!
JavaRanch » Java Forums » Databases » Hadoop
Bookmark "Decommission of Node" Watch "Decommission of Node" New topic
Author

Decommission of Node

jain Shasha
Greenhorn

Joined: Mar 20, 2013
Posts: 3
Hi

I want to understand how the decommission of a datanode happen on a hadoop cluster.
Also how much time does decommissioning would take.

Waiting for the reply.

Thanks.
sreekanth jella
Greenhorn

Joined: Aug 21, 2008
Posts: 10

From Hadoop definitive guide book:
The decommissioning process is controlled by an exclude file, which for HDFS is set by the dfs.hosts.exclude property and for MapReduce by the mapred.hosts.exclude property. It is often the case that these properties refer to the same file. The exclude file lists the nodes that are not permitted to connect to the cluster.

To remove nodes from the cluster:
1. Add the network addresses of the nodes to be decommissioned to the exclude file. Do not update the include file at this point.
2. Update the namenode with the new set of permitted datanodes, with this command:
% hadoop dfsadmin -refreshNodes
3. Update the jobtracker with the new set of permitted tasktrackers using:
% hadoop mradmin -refreshNodes

SCJP : 90%
jain Shasha
Greenhorn

Joined: Mar 20, 2013
Posts: 3
Thanks for the reply

Also how much does the decommission of a datanode took.
Because when i try to do the above steps it was taking a lot of time and i saw the same status Decommission in progress, It never changed to Decommissioned
Also what happen to the data store on the datanode (which we will be decommission) ?
sreekanth jella
Greenhorn

Joined: Aug 21, 2008
Posts: 10

The data that exist in this data node will be replicated by HDFS to available data nodes. I have not tried practically, but nodes should be decommissioned when the command "% hadoop dfsadmin -refreshNodes" is issued.
 
Consider Paul's rocket mass heater.
 
subject: Decommission of Node