If I want to connect to a particular website, I'll often use the URL class and it's openStream() method. However, on occasion, I can't connect to the web server this way as they probably have in place some sort of device to prevent the scanning from unknown sources. I can navigate fine to the same website using the exact same URL in my IE browser no problem. So I'm thinking that a browser must be identifying itself to the server somehow in the HTTPRequest object sent to the server. That being the case, how does one imitate the source request as having come from IE or some other browser? Is there a property I can set on some other class before I call the URL.openStream() method?
posted 3 years ago
Use the HtmlUnit library - it can imitate various browsers, and does a very good job of programmatic web access.