File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Java in General and the fly likes Write To A URL and Communicate With The Host Server Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Java in General
Bookmark "Write To A URL and Communicate With The Host Server" Watch "Write To A URL and Communicate With The Host Server" New topic
Author

Write To A URL and Communicate With The Host Server

JiaPei Jen
Ranch Hand

Joined: Nov 19, 2000
Posts: 1309
The java.net.URL is the package we use to read from and write to existing web pages. I have a Java program that imports the java.net.URL and reads and extracts some information from web pages. However, it is only the latter half of the task.
How does a Java program interact with; for example, cgi-bin scripts, on the server side provide that I have no idea about the contents of the cgi-bin scripts?
What I mean is that:
Many HTML pages contain forms-- text fields and other GUI objects that let us enter data to send to the server. After we type in the required information and initiate the query by clicking a button, the Web browser writes the data to the URL over the network. At the other end, a cgi-bin script (usually) on the server receives the data, processes it, and then sends us "a response", usually in the form of a new HTML page. As I stated in the beginning of this post, I have prepared a Java program to read the responding HTML page.

My question is regarding the first half of the process. How do I feed data in my Java program that initiate the query (instead of manually entering data in the text field) over the network for the host server to process? And then the Java program continues its work in reading the HTML page returned by the server? Provided that I do not know what kind of the program the hosting server uses to process the query.
gautham kasinath
Ranch Hand

Joined: Dec 01, 2000
Posts: 583
To make a request to a URL, URLConnection object.
Well, the same can also be used to submit data to the url. The simplest way would be to submit to the URL using query strings.. you know the "?name=..." kinda format after the url.
If there is anything more specific, please let me know.
Regds
Lupo


"In the country of the blind, the one eyed man is the King"
Gautham Kasinath CV at : http://www.geocities.com/gkasinath
John Smith
Ranch Hand

Joined: Oct 08, 2001
Posts: 2937

How do I feed data in my Java program that initiate the query (instead of manually entering data in the text field) over the network for the host server to process?

This is more or less straightforward:
JiaPei Jen
Ranch Hand

Joined: Nov 19, 2000
Posts: 1309
Thanks for your attention. The example can be viewed in the link below:
http://www.netsol.com/cgi-bin/whois/whois
At this web site, we can enter a domain name; say, yahoo.com, in the text field and click on the GO button. Then this HTML web page
http://www.netsol.com/cgi-bin/whois/whois?STRING=yahoo.com&SearchType=do&STRING2.x=32&STRING2.y=10
is returned by the host server. I am supposed to get the e-mail address of the Administrative Contact in that page.
I have to put the whole process in a Java program because I have to get the e-mail address of the Administrative Contact for a long list of domain names.
I have already prepared a Java program to retrieve the e-mail address of the Administrative Contact at the http://www.netsol.com/cgi-bin/whois/whois?STRING=yahoo.com&SearchType=do&STRING2.x=32&STRING2.y=10
I do not know the program that the host server uses to process the inquiry. My question is
1. how to feed domain names (of course one at a time in a loop) in a Java program for http://www.netsol.com/cgi-bin/whois/whois to receive as input? And
2. how do I know the URL of the HTML web page that is to be returned by the host server so that I can use my Java program to scan for the e-mail address of the Administrative Contact?
I am also thinking about logging into a remote web site that uses basic authentication or form based authentication and hold the session with a Java program.
Thanks in advance.
[ January 02, 2003: Message edited by: JiaPei Jen ]
[ January 02, 2003: Message edited by: JiaPei Jen ]
John Smith
Ranch Hand

Joined: Oct 08, 2001
Posts: 2937
I do not know the program that the host server uses to process the inquiry.

You don't need to know it whatsoever, all you need is an URL.

1. how to feed domain names (of course one at a time in a loop) in a Java program for http://www.netsol.com/cgi-bin/whois/whois to receive as input?

I already listed an example in the post above.

2. how do I know the URL of the HTML web page that is to be returned by the host server

You don't need to know that either, youjsut read the server response from the input stream using connection.getInputStream()

I am also thinking about logging into a remote web site that uses basic authentication or form based authentication and hold the session with a Java program.

This is also very straightforward, essentially you instantiate a class that incapsulates your user name and password, and a method of this class is called by your connection whenever needed:

Eugene.
[ January 03, 2003: Message edited by: Eugene Kononov ]
JiaPei Jen
Ranch Hand

Joined: Nov 19, 2000
Posts: 1309
Hello, Eugene, thanks for your responses to my questions. I tried you code; however, something gets screwed up:
http://www.netsol.com/cgi-bin/whois/whois
is the web site where I enter the query data in the text field. Suppose I enter; say, yahoo.com, and click on the GO button, I get this HTML web page with the link shown below in the address bar:

http://www.netsol.com/cgi-bin/whois/whois?STRING=yahoo.com&SearchType=do&STRING2.x=32&STRING2.y=10
And this is the page I am going to search the information I need.
I used your code; however, the post data are not written to the URL. I only get the "host web page". And I got the HTML code of that whole page printed to the console.
I experimented a little, and found that if I enter
http://www.netsol.com/cgi-bin/whois/whois?STRING=yahoo.com in the address bar, I am directed to an error page:
http://www.netsol.com/en_US/common/error.jhtml
and if I type http://www.netsol.com/cgi-bin/whois/whois?STRING=yahoo.com&SearchType=do
in the address bar, I will be able to get the expected web page.
Therefore, I must try to append &SearchType=do to the postData in the Java program. Nonetheless, the code shown below does not seem to work:

What is the correct way to do it?
[ January 03, 2003: Message edited by: JiaPei Jen ]
[ January 04, 2003: Message edited by: JiaPei Jen ]
[ January 04, 2003: Message edited by: JiaPei Jen ]
John Smith
Ranch Hand

Joined: Oct 08, 2001
Posts: 2937

postData += "?";

Your post data should start with the first parameter, not the question mark:
postData = "STRING=yahoo.com&SearchType=do"
Eugene.
[ January 05, 2003: Message edited by: Eugene Kononov ]
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Write To A URL and Communicate With The Host Server