Login to ASP page

[Sayed Ibrahim Hashimi]

I'm trying to automate a process of downloading a file from a website. The thing is that I have to fill in the username and password then click a button on this website. This webpage is using ASP. I have checked with the company maintaining the website and they don't have another protocol for downloading that file. I know this can be done in VB.NET, can this be done in Java?
Thanks.

[Peter den Haan]

I've got difficulties squaring "ASP" with "Java". What exactly are you trying to do?
- Peter

[James O'Dell]

I'm really interested in this too! I believe what Ibrahim is getting at is the following: Can one automate a remote ASP form using Java.
I have a customer that wants our company to upload files to their secure web server, however they believe that most businesses will have a person manually authenticate to their site, click a "browse" button to find a file on their local hard drive and then click "submit" to send the local file to the remote web server.
I was given the task of automating this. I believe the answer lies in using the HttpURLConnection class and formulating the proper query string, but I have not had success. Anyone done this before?
Thanks,
Jim

[Sayed Ibrahim Hashimi]

We'll here is how it is. Every week I have to go to a website then login. Then I have to mark a few checkboxes. Then I click the "Submit" button. This prompts the server to generate a file based on a database that they have. Then when that file has been generated, a URL to that file is returned then I download it. Now I'm trying to automate this.
I tried creating a URL with the parameters like this
www.thewebsite\login.asp?username=myusername&password=mypassword&check=on
But it doesn't seem to recognize those parameters. As a side note i took that same URL and placed it in a Browser and it doesn't make those settings. Maybe I'm incorrectly formatting it.
On top of that I have to "click" that submit button, how could I do this in Java?
Thanks in advance.

[Sayed Ibrahim Hashimi]

Originally posted by Ibrahim Hashimi:
We'll here is how it is. Every week I have to go to a website then login. Then I have to mark a few checkboxes. Then I click the "Submit" button. This prompts the server to generate a file based on a database that they have. Then when that file has been generated, a URL to that file is returned then I download it. Now I'm trying to automate this.
I tried creating a URL with the parameters like this
www.thewebsite.com\login.asp?username=myusername&password=mypassword&check=on
But it doesn't seem to recognize those parameters. As a side note i took that same URL and placed it in a Browser and it doesn't make those settings. Maybe I'm incorrectly formatting it.
On top of that I have to "click" that submit button, how could I do this in Java?
Thanks in advance.

As a side note, I have some VB.NET code that will do this but I can't use it on our system. Here is that code for those who know VB.NET (I don't ) This code will actually give fill in some textboxes for a date range whereas I am going to check a checkbox for all. Some of this info has been changed for security purposes.

Function GetFileFromPSI(Optional ByVal ReferenceDate As Date = #1/1/1900#) As String
      If ReferenceDate = #1/1/1900# Then
          ReferenceDate = Today
      End If
      Dim EndDate As Date = ReferenceDate.AddDays(1)
      Dim StartDate As Date = EndDate.AddDays(-15)
      Dim myCache As New CredentialCache()
      myCache.Add(New Uri("http://www.thewebsite.com/" :wink: , "Basic", New NetworkCredential("myusername", "mypassword"))
      Dim webAddress As String = "http://www.thewebsite.com/folder/Admin/export.asp";
      Dim webClientID As New System.Net.WebClient()
      webClientID.Credentials = myCache
      Dim parameterTable As New System.Collections.Specialized.NameValueCollection()
      With parameterTable
          .Add("txtFlag", "export123")
          .Add("txtStart", StartDate.ToShortDateString)
          .Add("txtEnd", EndDate.ToShortDateString)
          .Add("cboExportType", "EXAM")
      End With
      Dim StartPos, EndPos, FullPathLength As Integer
      Dim fileName, FullPath, LocalFileName As String
      LocalFileName = ""
     'Locate the file generated.
      Dim fileURI As String = "http://www.thewebsite.com/folder/Admin/exportfiles/";
      Try
          Dim responseArray As Byte() = webClientID.UploadValues(webAddress, "POST", parameterTable)
          Dim result As String = System.Text.Encoding.ASCII.GetString(responseArray)
          StartPos = result.ToLower.IndexOf(fileURI.ToLower)
          EndPos = result.ToLower.IndexOf(">download")
          FullPathLength = EndPos - StartPos
          FullPath = result.Substring(StartPos, FullPathLength)
          fileName = FullPath.Substring(fileURI.Length, FullPathLength - fileURI.Length)
          
          LocalFileName = StartDate.Year.ToString + "_" + StartDate.Month.ToString.PadLeft(2, "0") + _
                      "_" + StartDate.Day.ToString.PadLeft(2, "0") + "_" + fileName
          webClientID.DownloadFile(FullPath, LocalFileName)
      Catch
    'Handle error here
      End Try
      Return LocalFileName
  End Function

[Peter den Haan]

Clicking the submit button does no more than generating the request. The first problem is that the ASP server expects a POST rather than a GET request; the java.net.URL class will use GET by default. Try something like

HttpURLConnection uc = (HttpURLConnection)url.openConnection();
uc.setRequestMethod("POST");
uc.setDoOutput(true);
uc.setDoInput(true);
uc.connect();
// you can check response headers at this point,
// e.g. whether the response code is 2xx
//
// Next step is to send the form data over:
OutputStream os = uc.getOutputStream();
// write form fields to os, one line per field,
// as field=value pairs. The value must be encoded
// in x-www-form-urlencoded format (use
// URLEncoder.encode(String) for this)
os.close();
//
// Now you're ready to read the response:
InputStream is = uc.getInputStream);
// read the response from is
is.close();
uc.disconnect();

This is merely a skeleton for your code -- in real life, you'd want to check the response code, put the close() calls in finally clauses, use buffered streams, etc.
We're not there yet: from the VB.NET (ugh yuck splut) code it's clear that the site uses HTTP authentication. The code above remains unchanged except that you first install an Authenticator for the URL class to use:

Authenticator.setDefault(new Authenticator() {
   protected PasswordAuthentication getPasswordAuthentication() {
      return new PasswordAuthentication("username", "password");
   }
});

As you can see, a POST is a rather cumbersone operation, especially when authentication is involved. Many use helper classes, like Jason Hunter's excellent com.oreilly.servlet package. In this case, com.oreilly.servlet.HttpMessage would be suitable. It supports POST messages and BASIC authentication in a much more simple manner

HttpMessage msg = new HttpMessage(url);
msg.setAuthorization("username", "password");  
Properties form = new Properties();
form.put("field", "value");
InputStream in = msg.sendPostMessage(props);
// digest the response here
in.close();

I think you can see now why the O'Reilly helper classes are so popular
Hope this helps
- Peter

[Sayed Ibrahim Hashimi]

Peter
Thanks for those hints. I was unable to use the com.oreily.servlet package because I couldn't figure out how to authenticate to one URL then send the request to another. So I just decided to write the code from the "skeleton" that you gave me. I am able to do the POST and to get the results but when I try to decode the results with URLDecoder it always throws an IllegalArgumentException:

java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape
(%) pattern - "
at java.net.URLDecoder.decode(URLDecoder.java:167)
at java.net.URLDecoder.decode(URLDecoder.java:82)
at Export.test(Export.java:167)
at Export.main(Export.java:121)

Do you have any idea how I can resolve this issue? Thanks for all your help, it means a lot to me.

[James O'Dell]

That was a big help!
Thanks Peter

[Peter den Haan]

Originally posted by Ibrahim Hashimi:
[...] when I try to decode the results with URLDecoder it always throws an IllegalArgumentException:

Is the response encoded in the first place? Can you post the response text and/or locate where the decode function breaks exactly?
- Peter

[Sayed Ibrahim Hashimi]

Hi,
It seem like it is not decoded. If you haven't figured out I'm new to this whole network programming lol. I am able to get a response from the server, but it is not the response that I'm looking for. It just returns the same webpage I was viewing. I was thinking that I wasn't waiting for the server to crete the document so I tried using a BufferedReader and wait until reader.ready() is true but it will just return the same webpage. Then I wait after that again, and nothing happens. I even added connection.setFollowRedirects(true); to my code thinking that would solve my problem but it was to no avail. Any ideas? Thanks a lot.
Heres my code if you want to have a look

import java.net.*;
import java.util.*;
import java.io.*;
public class Test
{
public static void main(String args[])
{
    Test test = new Test();
    test.test();

}//end main()
public void test()
{
final char [] password = {'p','a','s','s','w','o','r','d' };       
try
{
Authenticator.setDefault( new Authenticator()
{
protected PasswordAuthentication getPasswordAuthentication()
{
return new PasswordAuthentication("myUserName",password);
}//end getPasswordAuth...
}//end anon class
);
URL psi = new URL("http://www.theWebsite.com/export/export.asp");
//note this URL is fake for security reasons
HttpURLConnection uc = (HttpURLConnection)psi.openConnection();
uc.setRequestMethod("POST");
uc.setDoOutput(true);
uc.setDoInput(true);
uc.setFollowRedirects(true);
uc.connect();
//conncetion should be successful at this point
//now set the parameters
OutputStream os = uc.getOutputStream();
                os.write( (URLEncoder.encode("exportType=EXAM" )).getBytes() );
os.write( (URLEncoder.encode("txtFlag=export123" )).getBytes() );
os.write( (URLEncoder.encode("txtStart=08/10/2002" )).getBytes() );
                os.write( (URLEncoder.encode("txtEnd=08/12/2002" )).getBytes() );
                //added this, but it made no diff.
                os.write( (URLEncoder.encode("cmdSubmit=Generate File")).getBytes() );
        os.flush();
os.close();
InputStream is = uc.getInputStream();
BufferedReader reader = new BufferedReader( new InputStreamReader(is) );
System.out.println("waiting");
while(true)
{
    if( reader.ready() )
break;
    else
System.out.print(".");
}
//now read the response
String line = "EMPTY";
while( line != null )
{
    if(reader.ready() )
    {
        line = reader.readLine();
        System.out.println(line);
                        //the response should have a file with a .dbf extension, just doing
                        //this to see if it worked. 
if(line.indexOf("dbf") != -1 )
    break;
    }
    else
    {
//System.out.print(".");
continue;
    }
}
        
}//end try
catch(Exception e)
{
e.printStackTrace();
}//end catch
}//end test
}//end class test

[ August 31, 2002: Message edited by: Ibrahim Hashimi ]

[Raghav Mathur]

hi
probably whatever you are trying to write to the server is not being sent to the server. That can be the only reason why it shows you the same page when you try to login . I,am stuck with the same prob . I,.am trying to capture the request from the client browser , make a Url connection for that request and read the contents .

Originally posted by Ibrahim Hashimi:
[QB]Hi,
I am able to get a response from the server, but it is not the response that I'm looking for. It just returns the same webpage I was viewing.
[ September 21, 2002: Message edited by: raghav mathur ]

[Raghav Mathur]

IN the above examples , the URl have been hardcoded . Suppose it is not then how does one judge whether to write a URL or read from it ?
Thanks in advance

[Peter den Haan]

Originally posted by raghav mathur:
IN the above examples , the URl have been hardcoded . Suppose it is not then how does one judge whether to write a URL or read from it ?

Depends on the HTTP method you invoke. A typical HTTP request opens a port to the HTTP server, sends the request header, a newline, and optionally a request body, then reads the response which will again consist of a header and an optional body.
The most often used methods are GET and POST; with GET, there is no request body and any parameters are encoded in the URL. With POST, the request body contains the parameter values submitted to the server. You can also upload files. For both GET and POST the server will typically respond with a response code and other header fields, followed by an HTML page.
- Peter

[Raghav Mathur]

Peter
so that means by guessing from the setRequestMethod()i can retrieve the whether pirtuclar request is using GET or POST .
(1)If the request is using GET ( without a query string) that means that the data has to be read from the URL.
(2)If the request is using GET "WITH" a query string that means data has to be writin to a URL .
(3)As far post is concerned , Data has to be writin in any case .
Is that so ?
Now suppose i have an application which acts as a proxy server between the client browser and the server . Whatever request is made by he client has to go through that proxy server application,The client makes a request , which connect to the proxy server aplication which then makes URL connection and get the request for the client .The proxy server appliation doesn't know what the clients request is but because the default is GET so that it is no prob for the proxy server application to make a URL connection and read the contents and deliver it to the client. Now if the client has to login to an ASP page using "POST" then the proxy application knows that the data has to be writin to the URL. If the client is using POST that means there would be no query string attched to it , then how would the proxy application know what it has to write to the URL? . how can be that info retrieved by the proxy application?
I hope i,am clear with my query .
Thanks in advance

[Peter den Haan]

Originally posted by raghav mathur:
(1)If the request is using GET (without a query string) that means that the data has to be read from the URL.
(2)If the request is using GET "WITH" a query string that means data has to be writin to a URL .

No. There is no difference between having a query string or not -- as I said, the method is significant. The GET method does not expect a request body, i.e. it does not expect you to write.

Now suppose i have an application which acts as a proxy server [...] Now if the client [uses] "POST" then the proxy application knows that the data has to be writin to the URL. If the client is using POST that means there would be no query string attched to it , then how would the proxy application know what it has to write to the URL?

The proxy would expect the client to write a request body with the POST, so it would capture whatever the client is writing and write that, in turn, to the web server. In the real world, rather than doing this in two discrete steps (read everything from the client, then write it to the server) a proxy would probably stream the write straight through to the remote server. In Java terms, it would write the bytes read from the client InputStream straight into the server OutputStream.
- Peter

[Raghav Mathur]

PETER

No. There is no difference between having a query string or not -- as I said, the method is significant. The GET method does not expect a request body, i.e. it does not expect you to write.

So then if an ASP page uses a GET method to actually write data on to the web server . The how will I judge whether to use GET or POST ? as you said that GET doen’t expect you to write.

In Java terms, it would write the bytes read from the client InputStream straight into the server OutputStream.

Aren’t these still 2 steps . The proxy in any case has to read from the client InputStream and then write it to the proxy servers OutputStream.
Anyways the biggest issue right now is how to judge whether to use GET ot POST.
Thanks in advance

[Raghav Mathur]

By reading the request header one can judge what request method to use. Now if i want to give my proxy server the functionality to handle all major protocols like socks,smtp,pop then what will be step i have to take .In case of post i know i have to write , in case of get i know i have to read , i will also need to use PUT ....This is about http . What if i have to handle Socks , ftp , smtp,pop etc .

Thanks in advance
[ October 16, 2002: Message edited by: raghav mathur ]

[Raghav Mathur]

Peter
I,am stuck with another prob now . I,am not able to extract the content length value from the header so that i can send it via post method to a specific URL( write to the URL) . What i was thinking of is that rather that extracting the content-length value( in case of post ) i should send the whole header ( which would include the content -length value) . What do you say ?
thanks in advance .
[ October 28, 2002: Message edited by: raghav mathur ]