aspose file tools
The moose likes Sockets and Internet Protocols and the fly likes web spider setting HTTP User-Agent Big Moose Saloon
  Search | Java FAQ | Recent Topics
Register / Login
JavaRanch » Java Forums » Java » Sockets and Internet Protocols
Reply Bookmark "web spider setting HTTP User-Agent " Watch "web spider setting HTTP User-Agent " New topic
Author

web spider setting HTTP User-Agent

Kevin Nilson
Greenhorn

Joined: Jun 17, 2002
Posts: 6
I am writing a web robot and I need for it to identify itself as a robot and not a regular web browser. This should be done by setting the User-Agent, but I can figure out how to do it.
Below is code similar to what I am doing. In the weblog it shows up as 'HTTP/1.1' for the user agent. I want it to say something like 'Java Program, not browser'.
URL url = new URL ("http://www.yahoo.com");
URLConnection connection = url.openConnection();
connection.setDoOutput(true);
BufferedReader in = new BufferedReader(
new InputStreamReader
(connection.getInputStream()));
String inputLine;
while ((inputLine = in.readLine()) != null)
System.out.println(inputLine);
in.close();

Thanks
Kevin Nilson


Kevin Nilson<br />climber@midwest.net<br />www.cs.siu.edu/~knilson
Mike Janger
Greenhorn

Joined: Nov 11, 2002
Posts: 1
Try this:
Properties props = new Properties(System.getProperties());
props.put("http.agent", "Kevin's non-browser Robot");
System.setProperties(props);
Mike Janger
Web Developer
Meridian Enterprises Corporation
[ November 11, 2002: Message edited by: Mike Janger ]
 
I agree. Here's the link: http://zeroturnaround.com/jrebel - it saves me about five hours per week
 
subject: web spider setting HTTP User-Agent
 
Similar Threads
HTTP and Cookies
java.io.IOException: Premature EOF
Passing cookies back and forth
Connecting to Internet from my Server
how to open a web page and checks the http responce