File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Sockets and Internet Protocols and the fly likes How far can you take HttpURLConnection Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Sockets and Internet Protocols
Bookmark "How far can you take HttpURLConnection" Watch "How far can you take HttpURLConnection" New topic
Author

How far can you take HttpURLConnection

Mark Mokris
Ranch Hand

Joined: May 08, 2002
Posts: 61
I have been experimenting and learning more about the use of HttpURLConnection. It looks like I can write Java apps which programmatically access web pages and web sites. I am just wondering how far I can take this.
I can do the equivalent of a HTML <form action="GET"...> thru an HttpURLConnection and read the results with a stream. What if the form script I am posting to is expecting an action="POST" rather than action="GET". Can I still accomplish this programmatically through the HttpURLConnection since I can't string together all the name/value pairs with the URL?
Can I use HttpURLConnection to "sign on" with my user and password to a website and continue through an entire "ASP session", just like I would if I was physically typing and interacting with the web site.
I have a specific application in mind, and the answers to these questions would be my first step.
Thanks!
Michael Morris
Ranch Hand

Joined: Jan 30, 2002
Posts: 3451
What if the form script I am posting to is expecting an action="POST" rather than action="GET". Can I still accomplish this programmatically through the HttpURLConnection since I can't string together all the name/value pairs with the URL?
Sure thing. You just have to understand the HTTP protocol. The form data in a POST request is encoded in the request body instead of the URL as in a GET.

Can I use HttpURLConnection to "sign on" with my user and password to a website and continue through an entire "ASP session", just like I would if I was physically typing and interacting with the web site.
Yep. Once again you have to fully understand the protocol and how and when to respond.
You can find (the boring but informative) white paper on HTTP at RFC 2616


Any intelligent fool can make things bigger, more complex, and more violent. It takes a touch of genius - and a lot of courage - to move in the opposite direction. - Ernst F. Schumacher
Mark Mokris
Ranch Hand

Joined: May 08, 2002
Posts: 61
Thanks a lot for that perspective Mike! I followed your link to the white paper on HTTP protocol at the W3C web site. I see how that is the authorative source on HTTP of course, but that doc is too tough for me. Are there any other sources for this information that might be a little more user friendly?
I just want to know how to construct some of the HTTP requests. Some examples would go a long way.
Michael Morris
Ranch Hand

Joined: Jan 30, 2002
Posts: 3451
Gettin' a little late for an old fart like me. I'll post (no pun intended ) some code for you in the morning. In the mean time take a look at the URLEncoder class if you are not already familiar with it.
Michael Morris
Ranch Hand

Joined: Jan 30, 2002
Posts: 3451
This is just a simple POST test that uses a test site: Test Form -- POST method
This code was adapted from an example by Cay Horstmann in Core Java 2 Volume II
HTTPPostTest.java

Here is the result of the post:
<HTML>
<HEAD>
<TITLE>CGI/1.0 test script report:</TITLE>
</HEAD>
<BODY>
<H1>CGI/1.0 test script report:</H1>
This CGI script was written in Perl 5 by <a href="mailto:wagner@mbox.cesnet.cz">
Z. Wagner</a>.<p>
<H2>General information</H2>
<UL>
<LI>SERVER_SOFTWARE = Apache/1.3.27 (OS/2) PHP/4.2.3 mod_ssl/2.8.11 OpenSSL/0.9.
6f
<LI>SERVER_NAME = hroch486.icpf.cas.cz
<LI>GATEWAY_INTERFACE = CGI/1.1
<LI>SERVER_PROTOCOL = HTTP/1.1
<LI>SERVER_PORT = 80
<LI>REQUEST_METHOD = POST
<LI>HTTP_ACCEPT = text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
<LI>HTTP_REFERER =
<LI>HTTP_USER_AGENT = Java/1.4.1
<LI>PATH_INFO =
<LI>PATH_TRANSLATED =
<LI>REQUEST_URI = /cgi-bin/echo.pl
<LI>SCRIPT_NAME = /cgi-bin/echo.pl
<LI>QUERY_STRING =
<LI>REMOTE_HOST =
<LI>REMOTE_ADDR = 24.204.13.72
<LI>REMOTE_PORT = 64670
<LI>REMOTE_USER =
<LI>AUTH_TYPE =
<LI>CONTENT_TYPE = application/x-www-form-urlencoded
<LI>CONTENT_LENGTH = 50
<LI>HTTP_COOKIES =
</UL>
<H2>Apache extensions</H2>
<UL>
<LI>SERVER_ADMIN = wagner@mbox.cesnet.cz
<LI>SERVER_SIGNATURE = <ADDRESS>Apache/1.3.27 Server at <A HREF="mailto:wagner@m
box.cesnet.cz">hroch486.icpf.cas.cz</A> Port 80</ADDRESS>
<LI>QUERY_STRING_UNESCAPED =
<LI>REDIRECT_QUERY_STRING =
<LI>REDIRECT_URL =
<LI>UNIQUE_ID = PrFMUpPniQIAAANGiLk
<LI>SCRIPT_FILENAME <i>is intentionally hidden (full file name of the script)</i
>
<LI>DOCUMENT_ROOT <i>is intentionally hidden</i>
</UL>
<H2>Query</H2>

<P><code>your_name=Michael+Morris&fruit=Apple&fruit=Banana

</code>
<H2>Parsed values</H2>
<UL>
</UL>
<H2>No cookies!</H2></BODY>
</HTML>
John Smith
Ranch Hand

Joined: Oct 08, 2001
Posts: 2937
Can I use HttpURLConnection to "sign on" with my user and password to a website and continue through an entire "ASP session", just like I would if I was physically typing and interacting with the web site.
Yes, even if it is a secure site. Essentially, any HTTP requests that you can do with your browser, you can do programmatically from the java program. As an example, I have a java program that connects to my online brokerage secure site via HTTPS, signs in as myself, continuously monitors the price action on a list of stocks, analyzes the price history, makes a trading decision (buy/sell), places the order, and logs the results of order execution, all without human intervention. All of this is done by java program "submitting" the forms, -- you simply add each name-value pair to your output stream, -- look how it is done in Michael's example.
So, yes, if you want to program a series of interactions with your browser, there is nothing really sacred about it, although it may seem so.
[ May 01, 2003: Message edited by: Eugene Kononov ]
Mark Mokris
Ranch Hand

Joined: May 08, 2002
Posts: 61
Thanks Michael and Eugene! Your posts are a tremendous help, and I appreciate the examples.
Mark
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: How far can you take HttpURLConnection