Grab Value from Remote Website

[Subliner Kemp]

Hi guys, I am developing a finance application where my application will need to grab rates (forex, etc) from various banks each day. These banks publish their rates at their website. I use Java application running generic InputBufferedStream of the IO library to grab the HTML data. Most websites work. Unfortunately, there is one, which is (www.pbebank.com) website, their server refuse to respond to my program. Despite adding User Agent equals Mozilla etc. to the header of my URLConnection, I still fail to get the required data. Would appreciate help if someone knows how to. I suspect it is still the server rejecting my application as the header sent still does not match up to their acceptable format. They are rejecting spiders/crawlers. My code is as below.

Error is, the page takes a long time run this function, and after a while, it returns:
java.net.SocketException: Software caused connection abort: recv failed

package bean;
/*
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import junit.framework.Assert;
*/
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.net.URLConnection;


public class pagescrape {
 
 public void grabUrl() {
    try {
      //final WebClient webClient = new WebClient();
      URL url = new URL("http", "www.pbebank.com", 80, "/en/en_content/personal/rates/index.html");
     // URL url = new URL("http", "www.rhb.com.my", 80, "/rate_charges/foreign-currency/fc_main.asp");
      
      String sHost = "www.pbebank.com";
      //String sHost = "www.rhb.com.my";
      
      String sReferer = "http://" + sHost;
    
      HttpURLConnection uc = (HttpURLConnection) url.openConnection();
      String sAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.18) Gecko/20110614 Firefox/3.6.18";
      
      System.setProperty("http.agent", sAgent);
      
      System.out.println(System.getProperty("http.agent"));
      
      uc.setRequestMethod("GET");
      uc.setRequestProperty("Host", sHost);
      uc.setRequestProperty("User-Agent", sAgent);
      uc.setRequestProperty("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8");
      uc.setRequestProperty("Accept-Language", "en-us,en;q=0.5");
      uc.setRequestProperty("Accept-Encoding", "gzip,deflate");
      uc.setRequestProperty("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
      uc.setRequestProperty("Keep-Alive", "115");
      uc.setRequestProperty("Connection", "keep-alive");
      uc.setRequestProperty("Referer", sReferer);
    
      //final HtmlPage page = (HtmlPage) webClient.getPage(url);
      //System.out.println(page.toString());
      
        BufferedReader in = new BufferedReader(
        new InputStreamReader(url.openStream()));

        String inputLine;
        while ((inputLine = in.readLine()) != null)
            System.out.println(inputLine);
        in.close();

      
    } catch (Exception exc) 
    {
      exc.printStackTrace();
    }
 }
}

[Steve Luke]

Subliner Kemp wrote:They are rejecting spiders/crawlers....

Your application is a crawler, and they are blocking crawlers. Have you read their terms of service to ensure that what you are doing is allowed? Either way, if they block you then there isn't much you can do about it.