wood burning stoves 2.0*
The moose likes Servlets and the fly likes Crawler Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login

Win a copy of Android Security Essentials Live Lessons this week in the Android forum!
JavaRanch » Java Forums » Java » Servlets
Bookmark "Crawler" Watch "Crawler" New topic


Sonal Jogi

Joined: Oct 19, 2004
Posts: 23

How to know if it is a web crawler??

Thanks in advance.
Jeanne Boyarsky
internet detective

Joined: May 26, 2003
Posts: 30130

A web crawler is a program that goes through web sites looking for links and then goes to those links and so on.

[Blog] [JavaRanch FAQ] [How To Ask Questions The Smart Way] [Book Promos]
Blogging on Certs: SCEA Part 1, Part 2 & 3, Core Spring 3, OCAJP, OCPJP beta, TOGAF part 1 and part 2
Stan James
(instanceof Sidekick)
Ranch Hand

Joined: Jan 29, 2003
Posts: 8791
Are you asking "How can you tell if the client requesting a page is a crawler or a browser?" I don't think you can. A crawler could be written to send headers that perfectly impersonate a given browser. I wonder if you could build the smarts to notice a particular session running up a lot of bandwidth in a short time.

A good question is never answered. It is not a bolt to be tightened into place but a seed to be planted and to bear more seed toward the hope of greening the landscape of the idea. John Ciardi
Don't get me started about those stupid light bulbs.
subject: Crawler
Similar Threads
Code for recursively downloading pages from particular web site(url)?
jdbc - common problems/mistakes
Art of Java
A Question on Lucene
<terminated, exit value: 0>C:\Program Files\Java\jre6\bin\javaw.exe (Feb 12, 2009 1:54:50 PM)