Java program to Login a website using url and to download a file

[Abhishekshri Shrivastava]

Hi,

Please help me to make a java program(basically i will be writing web apps) which will connect to an url, wil login to website and will download a spreadsheet from it.
Suggest me the feasibility and way to proceed.

Regards,
abhishek shrivastava

[Shashank Ag]

Abhishekshri Shrivastava wrote:
Please help me to make a java program(basically i will be writing web apps) ......

Well it will always be two step process unless you implement SSO (single sign on).
Here I am showing non-sso, simple approach:

1. Open login page on web browser and open html source of it.
The login page must be containing a form having input fields of username & password.
Note the form's action="<url>" and field names of username & password fields.

2. Now download apache HttpClient library and create instance of PostMethod with above url.
Also add parameters with above name and correct values and call PostMethod.send()

This will login your application into website.
NOTE: Take care to collect all cookies from response.

Now, similarly you call the url to download file but this time take care to add cookies collected above to request.

Hope it helps.

[Abhishekshri Shrivastava]

can you please share the sample code or reference material url for the suggested way.

[Shashank Ag]

can you please share the sample code or reference material url for the suggested way.

Here is a sample code:

import org.apache.commons.httpclient.Cookie;
import org.apache.commons.httpclient.HttpState;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.methods.GetMethod;

public class GetCookiePrintAndSetValue {

  public static void main(String args[]) throws Exception {

    HttpClient client = new HttpClient();
    client.getParams().setParameter("j_username", "abc");
    client.getParams().setParameter("j_password", "pqr");

    GetMethod method = new GetMethod("http://localhost:8080/");
    try{
      client.executeMethod(method);
      Cookie[] cookies = client.getState().getCookies();
      for (int i = 0; i < cookies.length; i++) {
        Cookie cookie = cookies[i];
        System.err.println(
          "Cookie: " + cookie.getName() +
          ", Value: " + cookie.getValue() +
          ", IsPersistent?: " + cookie.isPersistent() +
          ", Expiry Date: " + cookie.getExpiryDate() +
          ", Comment: " + cookie.getComment());
        }
      client.executeMethod(method);
    } catch(Exception e) {
      System.err.println(e);
    } finally {
      method.releaseConnection();
    }
  }
}

[Ulf Dittmer]

If this was my problem I'd use a higher-level library like jWebUnit that abstracts away a lot of the HTTP/HTML handling; it's easier to use and less code to write than if using HttpClient.

[Abhishekshri Shrivastava]

Ulf Dittmer wrote:If this was my problem I'd use a higher-level library like jWebUnit that abstracts away a lot of the HTTP/HTML handling; it's easier to use and less code to write than if using HttpClient.

If you can please share the idea in details

[Abhishekshri Shrivastava]

Shashank Ag wrote:

can you please share the sample code or reference material url for the suggested way.

Here is a sample code:

import org.apache.commons.httpclient.Cookie;
import org.apache.commons.httpclient.HttpState;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.methods.GetMethod;

public class GetCookiePrintAndSetValue {

  public static void main(String args[]) throws Exception {

    HttpClient client = new HttpClient();
    client.getParams().setParameter("j_username", "abc");
    client.getParams().setParameter("j_password", "pqr");

    GetMethod method = new GetMethod("http://localhost:8080/");
    try{
      client.executeMethod(method);
      Cookie[] cookies = client.getState().getCookies();
      for (int i = 0; i < cookies.length; i++) {
        Cookie cookie = cookies[i];
        System.err.println(
          "Cookie: " + cookie.getName() +
          ", Value: " + cookie.getValue() +
          ", IsPersistent?: " + cookie.isPersistent() +
          ", Expiry Date: " + cookie.getExpiryDate() +
          ", Comment: " + cookie.getComment());
        }
      client.executeMethod(method);
    } catch(Exception e) {
      System.err.println(e);
    } finally {
      method.releaseConnection();
    }
  }
}

I am using "commons-httpclient-3.0.1.jar" for it. is it ok? i am getting
"java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory
at org.apache.commons.httpclient.HttpClient.(HttpClient.java:65)
.........................................
......."

i have no idea what to do with "LogFactory" here
even i haven't used logfactory here(not making any logs).
please suggest me am i using correct jar
i am deploying it on weblogic 10.3

i also went through "org.apache.commons.httpclient.HttpClient" class from jar after decompling it.. but didn't get any proper idea

[Shashank Ag]

Well will need to use commons-logging.jar too in your class path.
Also you may need commons-codec*.jar, use it if your program gives error.

[Abhishekshri Shrivastava]

Shashank Ag wrote:Well will need to use commons-logging.jar too in your class path.

Also you may need commons-codec*.jar, use it if your program gives error.

How "commons-logging.jar " is used here
and what's the need of using "commons-codec*.jar"
please share me the link from which i can get all these jar
I am using jdk1.6

[Ulf Dittmer]

Abhishekshri Shrivastava wrote:i also went through "org.apache.commons.httpclient.HttpClient" class from jar after decompling it.

HttpClient is open source - if for some reason you want to check its source code you can just download it; no need to decompile. I'm guessing what you need are the javadocs and the tutorials, though, both of which are are on the HttpClient web site.

please share me the link from which i can get all these jar

They're all somewhere on commons.apache.org. If you're not yet familiar with that site I suggest you spend some time looking what's available - it has some very useful and popular libraries! See http://hc.apache.org/httpclient-3.x/dependencies.html

[Shashank Ag]

In simple terms HttpClient API is made using commons-logging and commons-codec api.
So, its dependent upon it and you need to have these in classpath while using HttpClient.

I am not attaching download links as you should be able to google it. Use codec 1.4 and logging 1.1 version.

[Abhishekshri Shrivastava]

Shashank Ag wrote:In simple terms HttpClient API is made using commons-logging and commons-codec api.
So, its dependent upon it and you need to have these in classpath while using HttpClient.

I am not attaching download links as you should be able to google it. Use codec 1.4 and logging 1.1 version.

I don't get any exception after adding thos jar files. but the thing is that how to proceed after setting paramater values as:
client.getParams().setParameter("login", "xyz");
client.getParams().setParameter("password", "xyz");
and then is it required to load the page or tell me how i will proceed to the downloadble link on welcome page after successfull login to the webpage.

and i had one more doubts reltaed to jsp and servlet I think you can also help me on the topic. please follow the link below:
http://www.coderanch.com/t/523495/JSP/java/setting-HTML-element-value-JSP

http://www.coderanch.com/t/523497/Servlets/java/set-JSP-HTML-element-values

http://www.coderanch.com/t/523492/JSP/java/Getting-name-JSP-servlet

[Shashank Ag]

Well to download once again using your web browser login to the site.

And upon the page where hyperlink to download the file is show stop there.
No right click on hyperlink and click copy link location (as it appears in my firefox) and paste it into notepad.

Now, in java program make a new HttpClient instance and call

client.getState().addCookies(<cookies[] of previous request>);
GetMethod method = new GetMethod("http://localhost:8080/");  
    try{  
      client.executeMethod(method);  
      InputStream in=method.getResponseBodyAsStream();
      FileOutputStream fos=new FileOutputStream(new File("test.txt"));
      byte kb[]=new byte[1024];
      int i=-1;
      while((i=in.read(kb))>-1) fos.write(i);
      fos.flush();
      fos.close();
    }

Your file should be created.
Note mostly the websites use cookies to determine authentication.
Hence, I asked you to firstly fetch cookies.

[Kavitha shan]

Hi Everyone,

I am in need of a java code to login to a website.

I tried the above given code, but am getting 'CANNOT FIND SYMBOL' error in place of GetMethod()

GetMethod method = new GetMethod("http://localhost:8080/");

Please help me.

Regards,
Kavitha