aspose file tools*
The moose likes General Computing and the fly likes Crawler Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Engineering » General Computing
Bookmark "Crawler" Watch "Crawler" New topic
Author

Crawler

Sandeep Jindal
Ranch Hand

Joined: Aug 25, 2003
Posts: 180
Hi,

Can anybody please tell me the difference between a web-crawler and a search engine. What I know is web-cralwer is a program that scans some web pages, follows the hyperlinks of the page and creates some index files related to the searched ones. But this is what a search engine too does . Please help me to get out of the confusion.

Regrads
Sandeep Jindal
[ March 09, 2005: Message edited by: Sandeep Jindal ]

SCJP 5.0
http://sites.google.com/site/duddlutechnologies/home
Dmitry Melnik
Ranch Hand

Joined: Dec 18, 2003
Posts: 328
Can anybody please tell me the difference between a web-crawler and a search engine. What I know is web-cralwer is a program that scans some web pages, follows the hyperlinks of the page and creates some index files related to the searched ones. But this is what a search engine too does .

You're right, search engine does it too.

I would tell that a search engine uses a crawler to build indexes. Or perhaps has it's own crawler-component.

The difference is that a search engine as well does many other things, which a crawler does not do. Like maintaining indexes, ranking, accepting search requests, running searches, sending out search results, you know...
Sandeep Jindal
Ranch Hand

Joined: Aug 25, 2003
Posts: 180
Hi Dmitry,

Thanks for your explanation.
So let me tell what i understand: A search engine has various components, and the biggest one is crawler whose responsibilty is to scan each web page, follow the hyperlinks and form index on that. Is this rite??

In other words, can I say crawler has no self identity as an application, that has to be integarated with some application like search engine?

Regards
Sandeep Jindal
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Crawler