inputStream openStream

[ilteris kaplan]

Hello Java friends,

I am struggling here with an issue which I couldn't figure out how to handle. Basically I am sending queries to google and returning links and then opening streams to each of these urls.

Here is my UrlReader class:

import java.io.*;
import java.net.*;
 
public class UrlReader {
private String urlPath;
private String content;

public UrlReader(String name) throws IOException {
urlPath = name;
readContent();
 
}

 
private void readContent() {
StringBuffer stuff = new StringBuffer();
try {
InputStream stream = openStream(urlPath);
BufferedReader reader = new BufferedReader(new InputStreamReader(stream));
String line;
while ((line = reader.readLine()) != null) {
stuff.append(line + "\n");
}
reader.close();
} catch (IOException e) {
e.printStackTrace();
}
content = stuff.toString();
}



 

private static InputStream openStream(String urlpath) {
InputStream stream = null;
try {
URL url = new URL(urlpath);
stream = url.openStream();
return stream;
}
catch (MalformedURLException e) {
System.out.print("Something is wrong with the URL: " + urlpath);
e.printStackTrace();
} catch (IOException e) {
System.out.print("There is a problem downloading from:  " + urlpath);
e.printStackTrace();
}
return stream;
}

public String getContent() {
return content;
}

}
 
 

the problem I am having is some urls are not responding or basically have 404 and when it happens my application stops running and quit with the message of forexample:

There is a problem downloading from: http://www.washingtonpost.com/wp-dyn/content/article/2006/03/23/AR2006032300817.htmljava.io.FileNotFoundException: http://www.washingtonpost.com/wp-dyn/content/article/2006/03/23/AR2006032300817.html
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:798)

what I want to do is build some kind of timer or a flag and if the connection fails or simply if the server doesn't respond I want my program to silently keep on crawling through the rest of the list of URLs.

What kind of way should I follow interms of that? I'd be grateful if you guys point me out the approach I should take here.


my best
ilteris

[ilteris kaplan]

is this a bad question to ask in this section?

thanks

[Ernest Friedman-Hill]

There's no way to know if a URL is bad until you try to open it. The immediate problem here is that if you find a bad URL and catch an exception, you return a null InputStream; then the "readContent" method tries to open a BufferedReader on that null stream, you get a NullPointerException, and the program aborts. If you catch an exception while opening a URL, don't continue trying to read it -- move on to the next one. The readContent() method could simply check for those nulls.

[ilteris kaplan]

thanks for the heads up! adding an if-clause did the trick.