File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
The moose likes Servlets and the fly likes Spiders???? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Servlets
Bookmark "Spiders????" Watch "Spiders????" New topic


Preethi Suryam
Ranch Hand

Joined: Nov 17, 2000
Posts: 92
Hi Everybody!
Can anybody throw some light on what is 'spider' all about?I know they r used in the concept of search engine.
Thanx for any reply.
Frank Carver

Joined: Jan 07, 1999
Posts: 6920
"spider" is effectively a synonym for "webcrawler". Both describe a piece of software which finds links in a web page and follows them to other web pages.
In the context of a search engine, imagine that you wish to set up a search engine for your own web site. For any kind of decent performance you really need an index, but there are several ways of creating that index. You could manually run some program on each web page, but you might forget a page, or forget to regenerate it when a page changes. You might also accidentally index a page that should not be publicly visible.
A better way might be to set some software pointing at your "home" page which indexes all the words on that page, then looks for all the links to other pages on your site and repeats the process. When it has finished it will have indexed all the publicly visible pages on your site.
You can then set this "indexing spider" to run from time to time (say first thing every monrning) so that whatever changes you make to the structure or content of your web site, the search index will always be up to date.

Read about me at ~ Raspberry Alpha Omega ~ Frank's Punchbarrel Blog
Preethi Suryam
Ranch Hand

Joined: Nov 17, 2000
Posts: 92
Hi Frank!
Thanx for ur i got a vague idea about spiders. what i understood about it is it's a program which tracks all the links and stores them, and finally we use those results for our search engine.i am sorry i was not able to get a more clear picture about this.anyway thanx once again.
Have a Nice Day!
I agree. Here's the link:
subject: Spiders????
It's not a secret anymore!