File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Crawler

 
Sonal Jogi
Greenhorn
Posts: 23
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

How to know if it is a web crawler??

Thanks in advance.
Sonal.
 
Jeanne Boyarsky
author & internet detective
Marshal
Posts: 33703
316
Eclipse IDE Java VI Editor
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sonal,
A web crawler is a program that goes through web sites looking for links and then goes to those links and so on.
 
Stan James
(instanceof Sidekick)
Ranch Hand
Posts: 8791
  • 0
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Are you asking "How can you tell if the client requesting a page is a crawler or a browser?" I don't think you can. A crawler could be written to send headers that perfectly impersonate a given browser. I wonder if you could build the smarts to notice a particular session running up a lot of bandwidth in a short time.
 
I agree. Here's the link: http://aspose.com/file-tools
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic