Debugging HttpURLConnection

[Dennis Putnam]

I am writing a post transaction for a less than ideally documented device. I can properly transmit post data via 'curl' so I know the interface works. However, when I try to post the same data via HttpURLConnect I get back a 415 (Unsupported media type) error. Here is the code segment I am using:

private void SendCommand(Map<String,String> command_) {
		sun.net.www.protocol.http.HttpURLConnection.level=ALL;
		StringJoiner commandString=new StringJoiner("&");
		for(Map.Entry<String,String> entry : command_.entrySet())
			try {
				commandString.add(URLEncoder.encode(entry.getKey(), "UTF-8") + "=" 
				     + URLEncoder.encode(entry.getValue(), "UTF-8"));
			} catch (UnsupportedEncodingException e) {
				// TODO Auto-generated catch block
				e.printStackTrace();
			}
		System.out.println(commandString.toString());
		byte[] out = commandString.toString().getBytes(StandardCharsets.UTF_8);
		int length = out.length;
		HttpURLConnection connection=getConnection();
		connection.setFixedLengthStreamingMode(length);
		connection.setRequestProperty("Content-Type", "application/x-www-form-urlencoded; charset=UTF-8");
		System.out.println(new String(out));
		try {
			connection.connect();
			try(OutputStream os = connection.getOutputStream()) {
				os.write(out);
			}
		}
		catch (IOException e) {
			// TODO Auto-generated catch block
			e.printStackTrace();
		}
		System.out.println(GetResults(connection));
	}

The 2 debug println commands display the proper post data but beyond that I have no clue what is being sent or where the problem is. I have found several references to debugging this this by setting:

-Djava.util.logging.config.file=logging.properties

As a VM argument in Eclipse. However, it also seems I need:

sun.net.www.protocol.http.HttpURLConnection.level=ALL

The problem is that I don't know where this goes. I tried adding it to the above code (line 2) but Eclipse tells me:

Multiple markers at this line
- ALL cannot be resolved to a variable
- level cannot be resolved or is not a field
- Access restriction: The type HttpURLConnection is not accessible due to restriction on required library C:\Program Files\Java
\jre1.8.0_91\lib\rt.jar

I'm guessing I am missing an import or maybe a library. Unfortunately the articles don't say what to import or where and Eclipse has no suggestions either.

Finally, it is not clear if this will really give me what I need to debug this. The article seems to imply that this will give me just the headers and not the full data transaction. I am looking for something that I can compare to the working 'curl -v' command. Can someone point me in the right direction? TIA.

[Ron McLeod]

This error usually indicates that the MIME type specified in the Content-Type header is not supported or is incorrect, or the HTTP header is not present.

Try using a network packet sniffer/analyzer such as Wireshark and compare what is going over the wire with your application and cURL.



If your client or server is running on Linux, you can also use tcpdump and direct the output to strings to take a quick and dirty look at your network traffic (although you will see some extraneous character displayed as data from layers below HTTP is rendered).

[root@tv-tb332 ~]# tcpdump -i eth1 -l -s 0 -nn port 88 -w - | strings
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes

PATCH /api/profile/Working/config/enb/cell-info HTTP/1.1
Host: 172.25.129.96:88
Connection: keep-alive
Content-Length: 33
User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36
Cache-Control: no-cache
Origin: chrome-extension://mkhojklkhkdaghjjfdnphfphiaiohkef
X-Ssi-Role: EXPERT
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip, deflate, sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,es;q=0.4,fr;q=0.2,ja;q=0.2
"cell-identity": 1268435

[Dennis Putnam]

Thanks for the reply. Unfortunately this is Windows and I was hoping to use Java to debug it. I guess I will need to install additional software (Wireshark).

[Ron McLeod]

Dennis Putnam wrote:.. I was hoping to use Java to debug it. I guess I will need to install additional software (Wireshark).

When I have issues with networked applications, I usually start by looking at the network layers and work my way up.

Also, you might want to try removing the charset=UTF-8 parameter from the Content-Type header and see if that changes anything. I've seen where where server applications don't properly recognise the content type when header value includes parameters.

[Dennis Putnam]

Well, I've managed to capture what I need but I don't really know what is required and what is not so I can't tell where the problem is. This is the packet dump for 'curl' which works:

=> Send header, 165 bytes (0xa5)
0000: 50 4f 53 54 20 2f 73 74 6f 6b 65 72 2e 50 6f 73 POST /stoker.Pos
0010: 74 5f 48 61 6e 64 6c 65 72 20 48 54 54 50 2f 31 t_Handler HTTP/1
0020: 2e 31 0d 0a 48 6f 73 74 3a 20 31 39 32 2e 31 36 .1..Host: 192.16
0030: 38 2e 30 2e 39 39 0d 0a 55 73 65 72 2d 41 67 65 8.0.99..User-Age
0040: 6e 74 3a 20 63 75 72 6c 2f 37 2e 34 38 2e 30 0d nt: curl/7.48.0.
0050: 0a 41 63 63 65 70 74 3a 20 2a 2f 2a 0d 0a 43 6f .Accept: */*..Co
0060: 6e 74 65 6e 74 2d 4c 65 6e 67 74 68 3a 20 32 33 ntent-Length: 23
0070: 0d 0a 43 6f 6e 74 65 6e 74 2d 54 79 70 65 3a 20 ..Content-Type:
0080: 61 70 70 6c 69 63 61 74 69 6f 6e 2f 78 2d 77 77 application/x-ww
0090: 77 2d 66 6f 72 6d 2d 75 72 6c 65 6e 63 6f 64 65 w-form-urlencode
00a0: 64 0d 0a 0d 0a                                  d....
=> Send data, 23 bytes (0x17)
0000: 74 61 41 46 30 30 30 30 31 31 30 41 37 44 46 36 taAF0000110A7DF6
0010: 33 30 3d 37 30 2e 35 

I couldn't figure out how to copy the reassembled TCP to the clipboard in Wireshark so I had to attach it as an image. As you can see there are definately differences but I don't know why or if it matters and where it does matter, I don't see how HttpURLConnection gives me any choices to change it. The order is obviously different and the host parameter is missing. Also 'Accept' is generic (*/*) for 'curl' but specific for Java. Does any of this help figure out what I'm doing wrong?

[Ron McLeod]

The curl message doesn't have a charset parameter in the Content-Type header.

[Dennis Putnam]

That was it. It works now, I owe you one. Thanks.