aspose file tools*
The moose likes I/O and Streams and the fly likes HTTP POST - does vanilla web page do URL encode? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » I/O and Streams
Bookmark "HTTP POST - does vanilla web page do URL encode?" Watch "HTTP POST - does vanilla web page do URL encode?" New topic
Author

HTTP POST - does vanilla web page do URL encode?

paul wheaton
Trailboss

Joined: Dec 14, 1998
Posts: 20532
    ∞

I am writing a functional test that will exercise a pretty vanilla HTTP servlet via POST.

I can test it just fine with GET, but now that I am implementing this test with POST, it just doesn't seem right.

I'm looking at online examples of using HttpURLConnection ...

does anybody know if the setRequestProperty is limited to stuff for the HTTP header? I was thinking of pushing in some app parameters ("first_name", "last_name", "password"...) there, but I suspect that that would be wrong.

I see people (on various java web sites) setting the content type: setRequestProperty("Content-Type","application/x-www-form-urlencoded"); But I'm thinking "do you have to?" Since the POST data payload doesn't go in the URL, then it would seem that we don't need to use that encoding. I would think that there would be some other way. Do servlets expect the parameter data coming in via POST to be URL encoded? Or, do they see that the payload is marked "application/x-www-form-urlencoded" and then choose to decode it before giving it to us?

I guess I imagine that there would be something like this:

HttpPost post = new HttpPost("http://localhost/myapp/register");
post.setParameter("username", "paul");
post.setParameter("password", "discowombat");
String response = post.send();

But ... I suppose no such luck?


permaculture Wood Burning Stoves 2.0 - 4-DVD set
Gregg Bolinger
GenRocket Founder
Ranch Hand

Joined: Jul 11, 2001
Posts: 15299
    
    6

HttpClient.
[ March 03, 2008: Message edited by: Gregg Bolinger ]

GenRocket - Experts at Building Test Data
paul wheaton
Trailboss

Joined: Dec 14, 1998
Posts: 20532
    ∞

Thanks for the link Gregg.

I'm cruising around their docs which are answering some of my questions. I'm pretty sure I'm not going to use this. Using a 2.8 meg library to replace six lines of code doesn't seem like the way I wanna fly with this.

I'm more in search of getting my head wrapped around this space. Looking for a little shop talk I guess.
paul wheaton
Trailboss

Joined: Dec 14, 1998
Posts: 20532
    ∞

Specifically, I'm finding this page to be useful.
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18541
    
    8

I'm using HttpClient to do HTTP POST as well. Works just fine. I really didn't want to learn all that HTTP detail when someone else had already figured it out for me. On the other hand I do have 46 jar files in my application's classpath...
paul wheaton
Trailboss

Joined: Dec 14, 1998
Posts: 20532
    ∞

I learned that the POST payload can be URLEncoded (apparently easier) or "multipart-mime". Reading more ...
paul wheaton
Trailboss

Joined: Dec 14, 1998
Posts: 20532
    ∞

More on the POST payload formats: http://www.webreference.com/html/tutorial13/4.html#HEAD-formenc
Gregg Bolinger
GenRocket Founder
Ranch Hand

Joined: Jul 11, 2001
Posts: 15299
    
    6

Originally posted by paul wheaton:
Using a 2.8 meg library to replace six lines of code doesn't seem like the way I wanna fly with this.


OTOH you'd be done.
Gregg Bolinger
GenRocket Founder
Ranch Hand

Joined: Jul 11, 2001
Posts: 15299
    
    6

I just looked at the jar file. It's 297.9 kb. Where are you getting 2.8 MB?
paul wheaton
Trailboss

Joined: Dec 14, 1998
Posts: 20532
    ∞

Originally posted by Gregg Bolinger:


OTOH you'd be done.


True!

This is something that is a lower priority than other things. I guess that as I fiddled with it, I was thinking "this seems clunkier than one would think." especially since the java libraries are usually pretty clean. On the occassion that they are skanky, you later discover that there are reasons of richness behind that first look goofiness. So .... is this something that should have been cleaned up, or is this something where there is something richer under the covers.

Since there is an apache library dedicated to this space, it would seem that the sun lib is probably a little weak.

Still, now I have my curiosity riled up. How is the post stuff conveyed? What is the format? Is it foolishness to fiddle with it? What would happen if I used setRequestProperty() for non-HTTP properties? Etc.
paul wheaton
Trailboss

Joined: Dec 14, 1998
Posts: 20532
    ∞

Originally posted by Gregg Bolinger:
I just looked at the jar file. It's 297.9 kb. Where are you getting 2.8 MB?


I downloaded the "bin".

Now that I'm looking in what I downloaded, I see there is a jar file in it that is 274k.

Still, I would rather just do the six lines of code for what I'm doing today.

It's good to know that this lib exists. I can think of far more interesting things that could be done with it.
Ben Souther
Sheriff

Joined: Dec 11, 2004
Posts: 13410

Originally posted by paul wheaton:

Still, I would rather just do the six lines of code for what I'm doing today.




Java API J2EE API Servlet Spec JSP Spec How to ask a question... Simple Servlet Examples jsonf
Ben Souther
Sheriff

Joined: Dec 11, 2004
Posts: 13410

Originally posted by paul wheaton:

I see people (on various java web sites) setting the content type: setRequestProperty("Content-Type","application/x-www-form-urlencoded"); But I'm thinking "do you have to?"


From these headers, it looks like URLConnection does that for you if you set
doOutput to true:

paul wheaton
Trailboss

Joined: Dec 14, 1998
Posts: 20532
    ∞

Ben,

Did you write that just now? I hope so. Because then you understand what I'm talking about. This is a simple little thing. Working in a library seems like overkill. And doing a clean, simple implementation is quite satisfying.

My version is very similar to yours (great minds?). I have try/finally blocks in mine. I didn't use BufferedReader cuz I know that my stuff is pretty small. So I just set up a byte array that is a meg, read that, got the size and made a string. (this is for functional tests, so throwing around a meg here and there doesn't bother me much - this isn't going into production)
Ben Souther
Sheriff

Joined: Dec 11, 2004
Posts: 13410

Originally posted by paul wheaton:
Ben,

Did you write that just now? I hope so. Because then you understand what I'm talking about. This is a simple little thing. Working in a library seems like overkill. And doing a clean, simple implementation is quite satisfying.

My version is very similar to yours (great minds?). I have try/finally blocks in mine. I didn't use BufferedReader cuz I know that my stuff is pretty small. So I just set up a byte array that is a meg, read that, got the size and made a string. (this is for functional tests, so throwing around a meg here and there doesn't bother me much - this isn't going into production)



I carved it out of a little load tester that I wrote a while ago.
Got rid of the threading and the try catches. Throwing Exception is enough for an example that does everything in the main method. It was assumed that you'd add them if you were doing more.


(great minds?)

Yes.... and us too.
[ March 03, 2008: Message edited by: Ben Souther ]
Peer Reynders
Bartender

Joined: Aug 19, 2005
Posts: 2922
    
    5
Originally posted by paul wheaton:
Since the POST data payload doesn't go in the URL, then it would seem that we don't need to use that encoding. I would think that there would be some other way.


Be careful where you are going with this.

RFC2616 Hypertext Transfer Protocol -- HTTP/1.1
4.2 Message Headers
4.5 General Header
5.3 Request Header Fields
6.2 Response Header Fields
7.1 Entity Header Fields

Any headers you add would typically classify as extension headers which are a subtype of entity headers:

Entity-header fields define metainformation about the entity-body or, if no body is present, about the resource identified by the request.


So you seriously have to ask yourself: Is that information you are stuffing into the header the actual payload or payload metainformation?

"Content-Type","application/x-www-form-urlencoded" makes it pretty clear that the parameters are the payload and that the parameters are "application/x-www-form-urlencoded" (payload metainformation). If you now go and move the payload into HTTP headers you are in effect subverting the HTTP protocol (And with REST awareness rising more people are going to be looking out for this). HTTP is an application protocol.

Is REST winning?

Joe Holloway: ...someone uses it to reinvent RPC once again. Even with the proprietary protocols I work with it seems like there's always someone ready to turn an application protocol into a transport protocol for their own application protocol.

[ March 04, 2008: Message edited by: Peer Reynders ]
Peer Reynders
Bartender

Joined: Aug 19, 2005
Posts: 2922
    
    5
Originally posted by paul wheaton:
I learned that the POST payload can be URLEncoded (apparently easier) or "multipart-mime". Reading more ...


HTML 4.01 Specification:
17.13.4 Form content types

application/x-www-form-urlencoded


"application/x-www-form-urlencoded" - payload metainformation
parameters - in the payload


multipart/form-data :


"multipart/form-data" - payload metainformation
parameters - in the payload

See the pattern?
Peer Reynders
Bartender

Joined: Aug 19, 2005
Posts: 2922
    
    5
Originally posted by paul wheaton:
I was thinking "this seems clunkier than one would think." especially since the java libraries are usually pretty clean. On the occassion that they are skanky, you later discover that there are reasons of richness behind that first look goofiness. So .... is this something that should have been cleaned up, or is this something where there is something richer under the covers.

Since there is an apache library dedicated to this space, it would seem that the sun lib is probably a little weak.


One possible limitation of the URLConnection class is its use of the (prefixed by Elliotte Rusty Harold with "moribund") java.net.ContentHandler class.

The other is that it is severely underdocumented. When I was trying to write a simple Java client for the RESTful Web Services sample bookmark application it took a lot of head-banging before I ran into some information that explained that URLConnection will automatically follow redirects and that I have to use setInstanceFollowRedirects(false) to suppress that. Following redirects makes sense for a web browser but in the context of a RESTful application the HTTP status code 301 "Moved Permanently" may simply mean that you just succeeded in permanently moving the resource through an UPDATE (it might also be of some idle interest to see if URLConnection will keep following the redirect and whether it could be trapped in a redirect loop).
 
It is sorta covered in the JavaRanch Style Guide.
 
subject: HTTP POST - does vanilla web page do URL encode?