This week's book giveaway is in the OO, Patterns, UML and Refactoring forum.
We're giving away four copies of Refactoring for Software Design Smells: Managing Technical Debt and have Girish Suryanarayana, Ganesh Samarthyam & Tushar Sharma on-line!
See this thread for details.
The moose likes Servlets and the fly likes Crawler Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


JavaRanch » Java Forums » Java » Servlets
Bookmark "Crawler" Watch "Crawler" New topic
Author

Crawler

Sonal Jogi
Greenhorn

Joined: Oct 19, 2004
Posts: 23
Hi,

How to know if it is a web crawler??

Thanks in advance.
Sonal.
Jeanne Boyarsky
author & internet detective
Marshal

Joined: May 26, 2003
Posts: 32481
    
214

Sonal,
A web crawler is a program that goes through web sites looking for links and then goes to those links and so on.


[OCA 8 book] [Blog] [JavaRanch FAQ] [How To Ask Questions The Smart Way] [Book Promos]
Other Certs: SCEA Part 1, Part 2 & 3, Core Spring 3, TOGAF part 1 and part 2
Stan James
(instanceof Sidekick)
Ranch Hand

Joined: Jan 29, 2003
Posts: 8791
Are you asking "How can you tell if the client requesting a page is a crawler or a browser?" I don't think you can. A crawler could be written to send headers that perfectly impersonate a given browser. I wonder if you could build the smarts to notice a particular session running up a lot of bandwidth in a short time.


A good question is never answered. It is not a bolt to be tightened into place but a seed to be planted and to bear more seed toward the hope of greening the landscape of the idea. John Ciardi
 
I’ve looked at a lot of different solutions, and in my humble opinion Aspose is the way to go. Here’s the link: http://aspose.com
 
subject: Crawler
 
It's not a secret anymore!