Retreiving information about a website ...

[Vassili Vladimir]

Hi,

I want to develop an application that returns information about a given input website.

As an example, if a user input a given website address to it, it needs to return information about that website.

Let's say that that the user enters www.google.com, in this case the application has to return some brief information about google.

Now the question is : Is that possible doing it with java, so i can open a socket on the site and read some given information ?

Are there any methodologies that let me do so ?

Thanks in advance ...

[Rob Spoor]

What kind of information do you want?

URLConnection and HttpURLConnection can help you out a bit:

URL url = new URL("http://www.google.com/");
URLConnection urlCon = url.openConnection();
System.out.println("Content-Type: " + urlCon.getContentType());
System.out.println("Content Length: " + urlCon.getContentLength());
if (urlCon instanceof HttpURLConnection)
{
    HttpURLConnection httpCon = (HttpURLConnection)urlCon;
    System.out.println("Response Code: " + httpCon.getResponseCode()); // e.g. 200 for OK, or 404 for not found
    System.out.println("Response Message: " + httpCon.getResponseMessage()); // 
}

[Vassili Vladimir]

The information i want to retrieve back is some kind of a description about the website itself, a textual description.

Are there any HTTP headers that get sent from the web-server back so i can read ?

Thanks ...

[Joe Ess]

I don't think HTTP headers are going to be useful to you.
You could try parsing META tags from the HTML source. They're used by convention to indicate key words to search engines. Of course, that means they may be just a list of keywords and not necessarily meaningful.
Your best bet may be to parse out the title tag, since it's human readable. Of course, both of these solutions rely on the HTML programmer to obey convention, and that is not really practical. For example, google.com has no meaningful META tags and the page title is simply "Google".
[ March 04, 2008: Message edited by: Joe Ess ]

[Vassili Vladimir]

Is there a parser to read the contents of the meta and title tags ?

Thanks ...

[Ulf Dittmer]

Is there a parser to read the contents of the meta and title tags ?

You could use something like HtmlTidy to clean up the HTML and hand it to you as XML (which makes it much easier to extract the parts you're interested in). A library like jWebUnit makes this even simpler.

But ultimately it's probably going to be fruitless, as the information you're looking is just not there, or rarely there.