Java with curl (http://curl.haxx.se/)

[Anil Karamchandan]

Hi,

I am not sure if this is the correct place to post this, please feel free if the moderators can find a better place for this post. I have a general question regarding using curl (http://curl.haxx.se/) with Java. I have a web-application running in tomcat, based on certain actions in my web-applications I need to write a java code which would trigger curl. Curl would then query a third-party application and return an XML/JSON.

this has to be read by me and return appropriate response to the user.

Can this be done ? I see that curl has a binary distribution for Linux, once I download it (http://curl.haxx.se/download.html) how do I call this from my java web-app.

thanks !

[Bear Bibeault]

Does it have to be curl? There are any number of ways to generate HTTP requests in Java.

[Anil Karamchandan]

I asked the same question to my boss and the answer was yes. Just for curiosity what other ways are you referring to ,I know httpConnection class can be used.

[Bear Bibeault]

There are the classes in java.net, and there are any number of 3rd-party implementations such as HttpClient.

[Anil Karamchandan]

thanks but how about doing it with curl ?

[Bear Bibeault]

Seems to me that calling out to a command line tool to do this is the worst possible approach.

But it looks as if there's a Java binding -- you might want to check that out.

[Anil Karamchandan]

Hi,

Looking at all the cases/possibilities, i have opted to do this using httpclient or urlconnection class. I have not written the complete code but have just manage to write which would get us started atleast..

If I had used curl - I would have to write the following.

curl -u username:password -d "param1=aaa& param2=bbb" -k http://www.testme.com/api/searches.xml

this would have returned me an output in XML

with httpclient i think this could be done with something like this...

import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.methods.PostMethod;

import java.io.BufferedReader;
import java.io.InputStreamReader;

public class HttpClientExampleTest
 {

    public static void main(String args[]) 
   {
      HttpClient httpclient = new HttpClient();
      BufferedReader bufferedreader = null;
       
      PostMethod postmethod = new PostMethod("https://www.testme.com/api/searches.xml");
      postmethod.addParameter("param1","aaa");
      postmethod.addParameter("params2", "bbb");
    
    try{
      int rCode = httpclient.executeMethod(postmethod);

      if(rCode == HttpStatus.SC_NOT_IMPLEMENTED) {
        System.err.println("The Post postmethod is not implemented by this URI");
        postmethod.getResponseBodyAsString();
      } else {
        bufferedreader = new BufferedReader(new InputStreamReader(postmethod.getResponseBodyAsStream()));
        String readLine;
        while(((readLine = bufferedreader.readLine()) != null)) {
          System.err.println(readLine);
      }
      }
    } catch (Exception e) {
      System.err.println(e);
    } finally {
      postmethod.releaseConnection();
      if(bufferedreader != null) try { bufferedreader.close(); 
} 
catch (Exception fe)
 {
	fe.printStackTrace();
 }
    }

  }
}

With this, I have only taken care of the parameters, how to I pass the username and password with this ? any help ?
Also by default, with CURL the username and password is encoded with base64 internally, that would mean now I have to take care of 2 things

1. Encode the Username and password with base64 format
2. Accommodate the username and password in httpclient call.

[Anil Karamchandan]

Ok, I was able to add authentication and encoding to the code, as of now the code looks something like this.

import org.apache.commons.codec.binary.Base64;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpStatus;
import org.apache.commons.httpclient.UsernamePasswordCredentials;
import org.apache.commons.httpclient.methods.PostMethod;

import java.io.BufferedReader;
import java.io.ByteArrayOutputStream;
import java.io.InputStreamReader;

public class NCS2
{

  public static void main(String args[]) {
   
    String username = "abc";
    String password = "xyz";

    HttpClient httpclient = new HttpClient();
    BufferedReader bufferedreader = null;
    
    
    PostMethod postmethod = new PostMethod("https://www.testabc.com/api/searches.xml");
    postmethod.addParameter("_def_id","8");
    postmethod.addParameter("dobday", "6");
    postmethod.addParameter("dobmonth","6");
    postmethod.addParameter("dobyear", "1960");
    postmethod.addParameter("firstname", "Test");
    postmethod.addParameter("lastname", "Test");
    
    
    String username_encoded = new String(Base64.encodeBase64(username.getBytes()));
    System.out.println("username_encoded ="+username_encoded);

    String password_encoded = new String(Base64.encodeBase64(password.getBytes()));
    System.out.println("password_encoded ="+password_encoded);

    httpclient.getState().setAuthenticationPreemptive(true);
    UsernamePasswordCredentials credentials = new UsernamePasswordCredentials();
    credentials.setPassword(username_encoded);
    credentials.setUserName(password_encoded);
    
   //httpclient.getState().setCredentials("", credentials);
    httpclient.getState().setCredentials("FORM","http://www.testabc.com/api/searches.xml",credentials);   // I am not sure which one to use here..

    try{
          int rCode = httpclient.executeMethod(postmethod);
          System.out.println("rCode is" +rCode);
      
          if(rCode == HttpStatus.SC_NOT_IMPLEMENTED) 
           {
              System.err.println("The Post postmethod is not implemented by this URI");
               postmethod.getResponseBodyAsString();
           } 
             else if(rCode == HttpStatus.SC_NOT_ACCEPTABLE) {
    	        System.out.println(postmethod.getResponseBodyAsString());
          }
           else {
                      bufferedreader = new BufferedReader(new InputStreamReader(postmethod.getResponseBodyAsStream()));
                       String readLine;
        while(((readLine = bufferedreader.readLine()) != null)) {
          System.out.println("return value " +readLine);
      }
      }
    } catch (Exception e) {
      System.err.println(e);
    } finally {
      postmethod.releaseConnection();
      if(bufferedreader != null) try { bufferedreader.close(); 
} 
catch (Exception fe)
 {
	fe.printStackTrace();
 }
    }

  }
  
 
}

Using this I get the return value for rCode as "406". Why am I receiving a response which is "Not Acceptable" Anything which would help me debug better and fix this.

[Rob Spoor]

First of all, the setCredentials method you're calling is deprecated. You should switch to the one that takes AuthScope and Credentials objects.

Secondly, I don't think you need to base64 encode the user name and password. I've checked the man page and found absolutely nothing about base64.

[Anil Karamchandan]

Hi Rob,

thanks for the reply, by authsope you meant, get the "realm" String using the authScope class ? I did saw that earlier but was not able to get more info about that ? can you please elaborate ?

thanks !

[Rob Spoor]

new AuthScope("www.testabc.com", 80)

will create an AuthScope instance for host www.testabc.com on port 80 for any realm.