Passed CCD-410 Hadoop Developer with 88%

[John Soper]

(Mods if you think this should be in the Hadoop forum, or both, please move)

I wouldn't say it was hard per se, but Hadoop has a lot of funky concepts.

For preparing, "24 hadoop interview questions" on the net is a great starting point. I then went through Tom White's Hadoop the Definitive Guide 3rd Ed, chapters 2-8 completely, typing out full notes and reading them constantly. For the other parts of the ecosystem, you need a high level overview with the basics of how they work and what they can do that Hadoop can't.

On my laptop, played with local-mode in Eclipse plus pseudo-mode with jar files. Tried lots of variations with a simple wordcount program:
Zero reducers, identity reducer, identity for both, combiners added
Logging and counter techniques
Made a chart of the 5-6 ways to set a property value and which took priority
List of the default conditions for things you don't specify (Hadoop can run given just IO paths).
Made a list of HDFS commands and kept playing around with them

Studied mock questions on the net, some had wrong answers, so be careful. Was going to purchase from HadoopExam.com, but they required submitting my MS Product Key, and I wasn't willing to do that. A shame, they advertised 200 questions for $45.

Took me a couple months of hardcore studying until it all became clear. The book took the most time, reading about 270 pages twice and doing notes.

In the future, I plan to also do the Cloudera certs for HBase, Hadoop admin, and Data Scientist. Will also take the Coursera Scala class with comes with a signed certificate. And maybe check into Apache Cassandra (no cert yet from what I could tell).

Big Data is fun.

[Dave Trower]

I am in the process of taking the Coursera class called "Introduction to Data Science". We got to practive hadoop on Amazon's platform.
You would probably like this class if it it comes around again.
Course Page

[John Soper]

Thanks for the heads up, I sure wish there were more hours in a day...

[Jhonson Ferndando]

Congratulations !

[Sv Balakrishnan]

Hi All,

I have also cleared the Hadoop Developer as well as Administrator exam in first attempt and cloudera exam should not be taken light, because of its cost as well as level of the questions. Please find below whatever I had followed to prepare the exam. However, I want to clarify here that www.HadoopExam.com do not ask any Microsoft (MS) keys for simulator. Once you install the trial version of their simulator on your computer, their software generate some unique key, which they need it to identify uniqueness of the machine to avoid any theft of the software and I don't see anything wrong with that.

I passed both CCAH & CCDH and here is what I did in that order.

1)www.HadoopExam.com Training Videos and Certification Simulator

Their Videos are very well designed for the core understanding of Hadoop Architecture. The lectures are very precise yet comprehensive, which cuts learning time significantly, besides providing a definite edge in certification exam and job interviews. Trainer did a great job in delivering the essential material in such a concise and effective manner that gives the learner very good foundation in Hadoop framework.

2)http://developer.yahoo.com/hadoop/tutorial/index.html

3)http://hadoop.apache.org/docs/current/

4)Hadoop Definitive Guide_Tom White

5)Hadoop Operations_Eric Sammer

6) Practice : Implemented Simple word count example small program to get hands on.

Thanks

[Muhammad Khojaye]

There is another course on Coursera named Web Intelligence and Big Data. Congratulations

[John Soper]

Muhammad Khojaye wrote:There is another course on Coursera named Web Intelligence and Big Data. Congratulations

I'm in the middle of the class now. It's pretty interesting, the teacher approaches things from a different angle and likes to discuss side issues. The first programming assignment was a simple map-reduce using a python framework, which is always a sweet language to play with.

Sorry about the bum steer on the product key, their website could be a little clearer though.

[Sudarshan Sreenivasan]

Can you share "Made a chart of the 5-6 ways to set a property value and which took priority " this ? ... I think it would be useful to a lot of people

[John Soper]

Sudarshan Sreenivasan wrote:Can you share "Made a chart of the 5-6 ways to set a property value and which took priority " this ? ... I think it would be useful to a lot of people

Sorry for the delay, just noticed this.

Five (sometimes six) ways to set configuration values (from highest priority to least):

   1.  In Java code with custom setter (which only some properties have)
				job.setNumReduceTasks(25)
		Getters also available
				System.out.println("numReducers: "+job.getNumReduceTasks());

   2.  In Java code with conf class before job creation
				conf.set("mapred.userlog.retain.hours", "23");
				conf.set("mapred.reduce.tasks", "100");
				Job job = new Job(conf, "movie count");

   3.  Set property on command line
  				hadoop jar hadoop-student.jar actorlist -D mapred.userlog.retain.hours=22 
					/user/hadoop/veggie/input /user/hadoop/veggie/output
				-D mapred.reduce.tasks=6
   	Repeat entry for multiple properties
				-D mapred.heartbeats.in.second=80 -D mapred.map.max.attempts=2
	Put value in single quotes if multiple words
				-D mapred.job.name='My Job'

   4.  Load custom configuration file on command line
				hadoop jar hadoop-student-homework-1.0.0-SNAPSHOT.jar actorlist 
					-conf /Users/john/custom.xml  /user/hadoop/veggie/input 
					/user/hadoop/veggie/output

   5.  Set environment variable to a dir which contains the three site-specific configuration files 
				export HADOOP_CONF_DIR=/Users/john/psuedo-conf

   6.  Change the "read-only" default configuration files under src/ 
		Bad idea because it affects all users, but possible (may get you walked out the door) 

[John Soper]

And passed Cloudera Hadoop Administrator today, again with 88%. That seems to be my number.

I definitely recommend trying to do them close together because there's a lot of common material.

The Cloudera practice questions help, even though they're $125.

Now onto studying functional programming, then data science after that. Already have pieces of both.

[Mahtab Alam]

Great Going Keep it up, Almighty is with you.

[Jayottam Jadhav]

Hi John Soper, i also wish to pass CCD-410 for which i have purchased practice test from cloudera. as you have mentioned that you also have taken some practice test from them and then have experience of giving real test. can you please guide me on following two question.

1) The questions form real test are from practice test ? YES/NO
if yes how many of them out of 60 approx.
2) Do i need anything other than this practice exam and explanation provided by them in result.

Thanks in advance.

[John Soper]

1. Hadoop admin is the only one where I'd bought the practice questions and took the cert. Don't remember if any were the same, but they were similar, and understanding the practice questions is helpful for your career in general. It's not just about passing the cert, you also need to answer interview questions.

2. Study Tom White's book over and over. Play with a simple word count program over and over, try lots of options.

3. Study for the Hadoop admin cert at the same time, lots of common ground. Only about 50% more work, but twice as impressive. Hadoop Operations is the book for that.

[danny wu]

Jayottam Jadhav wrote:Hi John Soper, i also wish to pass CCD-410 for which i have purchased practice test from cloudera. as you have mentioned that you also have taken some practice test from them and then have experience of giving real test. can you please guide me on following two question.

1) The questions form real test are from practice test ? YES/NO
if yes how many of them out of 60 approx.
2) Do i need anything other than this practice exam and explanation provided by them in result.

Thanks in advance.

Hi, Jayottam Jadhav and John Soper,
I am considering whether I need to buy the practice exam for CCD-410 and could you please let me know how many questions are included in the CCD-410 practice exam? my concern is that if the number of included practice questions is just 60, is it worth to spend 125 dollars for it?

thank you very much!

[saurabh pant]

Hi John Soper/Recent exam taker ,

I am also going for the CCD-410 in a few weeks. Can you please let me know nearly how many scenario based questions(code based questions) are going to be in the exam ? And on what topics ? Because in the hadoopexam dumps,they have got plenty of them.Some are them are even tough based on sequence file and some others.
Any help/advice is appreciated.

[Sai Babu]

Congratulations !! John and Bala,

I am also preparing for developer and admin exams, i am a newbie though,

i'd like to know if there is any document on how to install the sand box for practicing.

any help is greatly appreciated!!

[John Soper]

Don't know about any sandbox tools. I still think the best way is to master (or even memorize) word count code, everything else is built from that. My two favorite examples are below. Besides that, Tom White and Eric Sammer are your new best friends, don't go anywhere without their books.

Streaming:

  export STREAM_JAR=/usr/local/Cellar/hadoop/1.0.4/contrib/streaming/hadoop-streaming-*.jar
  hadoop jar $STREAM_JAR -input ./input -output ./output -mapper ./mapper.sh -reducer "uniq -c"

  mapper.sh contents:
	cat | tr ' ' '\n' | tr -d '[:punct:]' | tr '[:upper:]' '[:lower:]' 

Non-Streaming:

unset HADOOP_CONF_DIR
export HADOOP_CORE_JAR=/usr/local/Cellar/hadoop/1.0.4/hadoop-core-*.jar
mkdir wordcount_classes
javac -classpath $HADOOP_CORE_JAR -d wordcount_classes WordCount.java
jar -cvf ./wordcount.jar -C wordcount_classes/ .
hadoop jar ./wordcount.jar org.myorg.WordCount ./input ./output


File WordCount.java:

package org.myorg;
        
import java.io.IOException;
import java.util.*;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapreduce.*;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
        
public class WordCount {
        
 public static class Map extends Mapper<LongWritable, Text, Text, IntWritable> {
    private final static IntWritable one = new IntWritable(1);
    private Text word = new Text();
        
    public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);
        while (tokenizer.hasMoreTokens()) {
            word.set(tokenizer.nextToken());
            context.write(word, one);
        }
    }
 } 
        
 public static class Reduce extends Reducer<Text, IntWritable, Text, IntWritable> {

    public void reduce(Text key, Iterable<IntWritable> values, Context context) 
      throws IOException, InterruptedException {
        int sum = 0;
        for (IntWritable val : values) {
            sum += val.get();
        }
        context.write(key, new IntWritable(sum));
    }
 }
        
 public static void main(String[] args) throws Exception {
    Configuration conf = new Configuration();
    Job job = new Job(conf, "wordcount");
    
    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(IntWritable.class);
        
    job.setMapperClass(Map.class);
    job.setReducerClass(Reduce.class);
        
    job.setInputFormatClass(TextInputFormat.class);
    job.setOutputFormatClass(TextOutputFormat.class);
        
    FileInputFormat.addInputPath(job, new Path(args[0]));
    FileOutputFormat.setOutputPath(job, new Path(args[1]));
        
    job.waitForCompletion(true);
 }
}

[zach zhang]

Hi john:

Could you share you daily time schedule for preparing for the exams? say how many hours per day and how many days in total?

thanks!

[John Soper]

That's a pretty tough question. Maybe 4-6 man-weeks for Developer, then 2-3 man-weeks for Admin (if developer was just passed). Having the slides from training class and the example exams will help reduce that some.
If you go into this field, you're going to have to learn that stuff anyway though. It's not like it's a diversion takes you away from something really important.
Here's the notes for the Admin cert. I saved a copy and cleaned up the editing and added a few things as I went through Hadoop Operations
notes

[Sai Babu]

Hey John,

appologies i did'nt check this forum for quite some time.
Thanks for the suggestions i already made Tom White my best buddy
i started read Hadoop: The Definitive guide, it seems to be great source for Hadoop.

yet to meet my another new friend Eric Sammers,

Thanks guys & best of luck in preparing for this exam.

[Jhon Anderson]

Kudos mate. Even I'm planning to do a certification in Hadoop.

[Ruchi Saini]

I have recently passed ccd-410, have a look at my blog https://ruchisaini.wordpress.com/2015/05/30/cloudera-hadoop-developer-certificationccdh/ ...i have written my points here

[Sith Michel]

Its a similar questions that I raised here https://goo.gl/nJqPRG and getting the various suggestions. If you have some more good suggestions you can also give..

[Vidya Shivram]

Hi all , do we have coderanch Hadoop cerification forum ? If yes, please provide me the link or guide me to right hadoop certification forum

[Jeanne Boyarsky]

Vidya Shivram wrote:Hi all , do we have coderanch Hadoop cerification forum ? If yes, please provide me the link or guide me to right hadoop certification forum

For now, please post in the product & other certs forum.  If we see a few posts, we can create a forum and move them into it.

[Harri son]

I’m usually to blogging and i actually appreciate your content. The article has genuinely peaks my interest. I’m going to bookmark your web page and maintain checking for new details.