Extract data from Hadoop File system using nutch

[syruss kumar]

Hi,

I’m newbie to nutch.I have installed and configured nutch to crawl the site.I want to extract the data from the crawl db .Is there any way to get the data programmatically?

Thanks in advance,

[syruss kumar]

Hi all,

Here is the solution. Use Nutch api to extract the data.Under crawl/segment folder it placed the content,parsed text,parsed data etc.

Sample code to read data from hadoop file system using Nutch 1.6 api

	public Content readSegment(String segPath,String url){
		
		Text key= new Text(url);
		Path path= new Path(segPath);
		Content content = null;

		ArrayList<Writable> parsedLst = null;
		Map<String,List<Writable>> results=new HashMap<String, List<Writable>>();
		SegmentReader reader= new SegmentReader(configuration,true,true,true,true,true,true);
		try {
			reader.get(path, key, new StringWriter(), results);
			parsedLst=(ArrayList<Writable>) results.get("co");
			Iterator<Writable> parseIter=parsedLst.iterator();
			while(parseIter.hasNext()){
				content=(Content) parseIter.next();
			}
		} catch (Exception e) {
			e.printStackTrace();
		}

		return content;
	}

[parin jogani]

Thank you! of great help.
Any way to extract a particular file format only (eg. pdf)?