This week's book giveaway is in the OO, Patterns, UML and Refactoring forum. We're giving away four copies of Refactoring for Software Design Smells: Managing Technical Debt and have Girish Suryanarayana, Ganesh Samarthyam & Tushar Sharma on-line! See this thread for details.
Are you asking "How can you tell if the client requesting a page is a crawler or a browser?" I don't think you can. A crawler could be written to send headers that perfectly impersonate a given browser. I wonder if you could build the smarts to notice a particular session running up a lot of bandwidth in a short time.
A good question is never answered. It is not a bolt to be tightened into place but a seed to be planted and to bear more seed toward the hope of greening the landscape of the idea. John Ciardi
I’ve looked at a lot of different solutions, and in my humble opinion Aspose is the way to go. Here’s the link: http://aspose.com