my dog learned polymorphism
The moose likes Beginning Java and the fly likes Getting web pages from an app. Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login

Win a copy of Make it so: Java DB Connections & Transactions this week in the JDBC forum!
JavaRanch » Java Forums » Java » Beginning Java
Bookmark "Getting web pages from an app." Watch "Getting web pages from an app." New topic

Getting web pages from an app.

Bob Burnquist

Joined: Dec 12, 2005
Posts: 13
My question is, what is the best way to call a number of web pages from a java class.

I put this in the servlets section but then realisied my code does not need to be a servlet as this application will be called by a cron job.

It needs to go through a list and make requests to a lot of websites so that the urls can be cached server-side. Im not interested in rendering any of these websites, as long as the pages are sent from the server.

This easy to do through sockets?

Thanks loads.
[ March 10, 2006: Message edited by: Sebastian Patten ]
Vishnu Prakash
Ranch Hand

Joined: Nov 15, 2004
Posts: 1026
Looks more appropriate to in servlet forum.

Servlet Spec 2.4/ Jsp Spec 2.0/ JSTL Spec 1.1 - JSTL Tag Documentation
Stan James
(instanceof Sidekick)
Ranch Hand

Joined: Jan 29, 2003
Posts: 8791
If you just want a Java client that can pull pages from web sites and write them to disk it's pretty simple to do. Look into the URL class which can get you a URLConnection. From there you get the stream and read whatever is at that address. For HTML you can probably use a buffered reader with no problem. For images or other binary files stick with binary streams.

Take a look at the doc and see what you can put together. If you get stuck, show us some code and we'll try to get you going again.

Keep it fun!

A good question is never answered. It is not a bolt to be tightened into place but a seed to be planted and to bear more seed toward the hope of greening the landscape of the idea. John Ciardi
I agree. Here's the link:
subject: Getting web pages from an app.
It's not a secret anymore!