File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Web Crawler in Java

 
Himanshu Gupta
Ranch Hand
Posts: 598
Android Eclipse IDE Ubuntu
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have a task in which I have to scan the HTML source of web pages and extract some information depending upon the pattern. The extracted information will be save in the database for the business purpose. The amount of data being extracted in not much.

I am searching for an appropriate web crawler written in java.
Does anyone has some suggestion to give or any other inputs to be share.

Any kind of help is highly appreciated.
 
Lester Burnham
Rancher
Posts: 1337
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You can find crawlers at www.java-source.net. For extracting information, use a library like HtmlUnit.
 
Himanshu Gupta
Ranch Hand
Posts: 598
Android Eclipse IDE Ubuntu
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Lester for the reply. I have selected few crawlers projects and have decided to try them making a small prototype of each.
 
Michael Ergotron
Greenhorn
Posts: 5
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I use httpclient for similar purposes...

Another option is to call wget command from java. It works fine when you have https
 
Vlada Stankovic
Greenhorn
Posts: 14
Java MySQL Database Spring
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Its better to use perl for this kind of tasks. You have mechanize or LWP library on CPAN. HTMLUnit is too slow.
 
David Davidov
Greenhorn
Posts: 1
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I think so you can try php examples firs, which you can find here:

http://broobee.com/folder/12/crawlers-applications
 
I agree. Here's the link: http://aspose.com/file-tools
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic