File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
The moose likes Beginning Java and the fly likes Getting web pages from an app. Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Beginning Java
Bookmark "Getting web pages from an app." Watch "Getting web pages from an app." New topic

Getting web pages from an app.

Bob Burnquist

Joined: Dec 12, 2005
Posts: 13
My question is, what is the best way to call a number of web pages from a java class.

I put this in the servlets section but then realisied my code does not need to be a servlet as this application will be called by a cron job.

It needs to go through a list and make requests to a lot of websites so that the urls can be cached server-side. Im not interested in rendering any of these websites, as long as the pages are sent from the server.

This easy to do through sockets?

Thanks loads.
[ March 10, 2006: Message edited by: Sebastian Patten ]
Vishnu Prakash
Ranch Hand

Joined: Nov 15, 2004
Posts: 1026
Looks more appropriate to in servlet forum.

Servlet Spec 2.4/ Jsp Spec 2.0/ JSTL Spec 1.1 - JSTL Tag Documentation
Stan James
(instanceof Sidekick)
Ranch Hand

Joined: Jan 29, 2003
Posts: 8791
If you just want a Java client that can pull pages from web sites and write them to disk it's pretty simple to do. Look into the URL class which can get you a URLConnection. From there you get the stream and read whatever is at that address. For HTML you can probably use a buffered reader with no problem. For images or other binary files stick with binary streams.

Take a look at the doc and see what you can put together. If you get stuck, show us some code and we'll try to get you going again.

Keep it fun!

A good question is never answered. It is not a bolt to be tightened into place but a seed to be planted and to bear more seed toward the hope of greening the landscape of the idea. John Ciardi
I agree. Here's the link:
subject: Getting web pages from an app.
It's not a secret anymore!