Hadoop(Beginner Level Question)

[Supraja Jayakumar]

Hi

I wrote this piece of hadoop to preprocess files and write the files again to the output directory. I see files by name part-000, 0001 and so on being created but they all are empty. I use NullWritable for key. But set Text for value. I am not sure if its because of that.

The following is my code:

public class Preprocessor {

public static void main(String[] args) throws Exception {

    String inputFiles = args[0];
    String outputDir = args[1];

    Configuration con = new Configuration();
    JobConf conf = new JobConf(con, Preprocessor.class);

    FileInputFormat.setInputPaths(conf, inputFiles);
    FileOutputFormat.setOutputPath(conf, new Path(outputDir));

    conf.setJobName("Preprocessing Query Data");
    conf.setMapperClass(Map.class);
    conf.setInputFormat(TextInputFormat.class);
    conf.setOutputKeyClass(NullWritable.class);
    conf.setOutputValueClass(Text.class);
    conf.setOutputFormat(TextOutputFormat.class);
//      conf.setNumMapTasks(500);
    conf.setNumReduceTasks(0);

    JobClient.runJob(conf);
        }
}
Map class:

public void map(LongWritable key, Text value, OutputCollector<NullWritable, Text>   
               output,Reporter reporter) throws IOException {

    String input = value.toString();
    String[] inputLines = input.split("\n");
    int numInputLines = 0;
    while(numInputLines != inputLines.length) {
        String[] queryComponents = inputLines[numInputLines].split("\\s+");
        String originalQuery = queryComponents[1];
        if(originalQuery.contains("[A-Za-z]")) {
            String newQuery = queryComponents[0] + "    " + 
                            originalQuery + "\n";
            collector.collect(NullWritable.get(), new Text(newQuery));
        }
        numInputLines++;
    }
}

[Alan Gates]

TextInputFormat already splits your input by line and only hands your map function one line at a time. So you don't need the while loop. Also, String.contains() takes a CharSequence, not a regular expression. Unless you are looking for the literal character sequence "[A-Za-z]" you want to use String.matches().