Compressed data from request

[Ricardo Carvalho]

Hi all. I call a webservice in java and i get a response but it comes garbled. Someone said it may be compressed with gzip. How do i uncrompress it?
Thanks

[Ron McLeod]

Check the HTTP response headers so that you know what you are getting.

Content-Type: what kind of content/resource is being returned (before any encodings) - text, image, binary blob, etc.
Content-Encoding: what kind of encoding (typically compression) have been applied to the content - gzip, deflate, etc.
Transfer-Encoding; similar to Content-Encoding, but applied on a hop-by-hop basis; may also indicate that the content is being transferred in chunks

[Ron McLeod]

Also, decompression is often done in the HTTP client library for you.  Which library are you using?

[Ricardo Carvalho]

Ron McLeod wrote:Also, decompression is often done in the HTTP client library for you.  Which library are you using?

Spring i think.. and JDK 17
The content says gzip

[Ron McLeod]

Ricardo Carvalho wrote:Spring i think..

You probably need to sort that out to move forward.

If you can share some example code, maybe someone can figure out what you are using.

[Ricardo Carvalho]

Ron McLeod wrote:

Ricardo Carvalho wrote:Spring i think..

You probably need to sort that out to move forward.

If you can share some example code, maybe someone can figure out what you are using.

Yes, i was writting on mobile phone.
Just for a little context, i'm calling an API from a game, and this is what i call:
  http://api.planets.nu/games/list?username=wasp
If i put that on the browser i get a json:
[{"name":"Athos IV Sector Campaign","description":"No wasp, Max advantage 600, planets limit ships 20\/1\/0, fog of war, blinded scores, random spaced home worlds, low minerals. Allies 0, Share Intel 1, Safe Passage 1. Host 2\/week.","shortdescription":"Custom Game","status":2,"datecreated":"5\/12\/2023 12:45:03 PM","dateended":"1\/1\/0001 12:00:00 AM","maptype":3,"gametype":2,"wincondition":1,"difficulty":1.9214988730278,"tutorialid":0,"requiredlevelid":7,"maxlevelid":0,"masterplanetid":172,"quadrant":0,"mintenacity":0,"faststart":10,"turnsperweek":0,"yearstarted":149,"isprivate":false,"scenarioid":0,"createdby":"smi","turn":15,"slots":11,"turnstatus":"1_34_6789AB","hostdays":"__T__F_","slowhostdays":"__T__F_","hosttime":"22:00","lastbackuppath":"d:\\planetsdata\\backups\\game563278\\turn14-638409052648798010.zip","nexthost":"1\/19\/2024 10:00:00 PM","allturnsin":fals......

If i do this code:

package com.example.consumingrest;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.web.client.RestTemplateBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Profile;
import org.springframework.web.client.RestTemplate;


@SpringBootApplication
public class ConsumingRestApplication {

private static final Logger log = LoggerFactory.getLogger(ConsumingRestApplication.class);

public static void main(String[] args) {
SpringApplication.run(ConsumingRestApplication.class, args);
}

@Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
return builder.build();
}

@Bean
@Profile("!test")
public CommandLineRunner run(RestTemplate restTemplate) throws Exception {
return args -> {
String ret  = restTemplate.getForObject(
"http://api.planets.nu/games/list?username=wasp",String.class);

log.info("RESULTADO:   " + ret);

};
}

}

I get this

 : RESULTADO:    $K�e ��Vko�6�+w��9zX�$
� ���4���.Z�-���T\o��!��y��2`� �0��<�<���$�xP�Q�^�FW��J�%�z&v���J��
%a�Zў�>�W����I�v�QR�2ɭ�Vt’iDo(�Vg���[�#���i�
Y�L�47!i&kՑ�Y��Fu��J�5�Z��NH�YkftѶ�B���iN/��-�X�-���WM<�_����������(m�r9���8``�LP&�Y^i��f@�$JR��r��QJo_�w}�*ی��E>t�:��C��0�!ո@��������Nl��Z{�rV$���e

If i try with python, it uncompress automatically
Is this enough? what else do you need?

Thanks

[Ricardo Carvalho]

I should have posted the real output. sorry.
Oh, and i'm using InteliJ

[Tim Holloway]

"Real" output for us would be preferably text cut-and-paste, not screenshots. Screenshots should be avoided as they eat up storage space on our servers, are often hard to read at different user's screen resolutions and don't show up for reference when we're editing our replies.

Another thing that makes things more readable for us is if you use Code tags to wrap pre-formatted code such as Java source, XML or SQL.  The "Code" button on the message editor will create Code tags for you.

Yes, that server says that it will accept gzip-compressed data requests and it has returned a gzip-compressed data response. So you need to ensure that either the method you used to make the ReST request has built-in gzip decompression turned on (if available), or alternatively, gunzip the data yourself.

[Ricardo Carvalho]

Sorry. Didnt know those rules. Will watch out next time.
About the issue, i have tried to put on file but gunzip wont recognise it.
Do you have a java example that works? It can be any library.
Thanks

[Ron McLeod]

This is what the response headers look like:HTTP/1.1 200 OK Cache-Control: private Content-Type: text/javascript; charset=UTF-8 Content-Encoding: gzip Vary: Accept-Encoding Server: Microsoft-IIS/8.5 Access-Control-Allow-Origin: * Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept X-AspNet-Version: 4.0.30319 X-Powered-By: ASP.NET Date: Wed, 18 Jan 2024 17:51:04 GMT Content-Length: 1341
I would expect that Spring RestTemplate client API/library supports automatically decompressing gzip'ed content for you, but I don't use Spring, so I don't know if this is something that you need to configure (or even if it is available at all).  Hopefully someone that works with Spring can help answer that.

In the non-Spring environments that I have worked with, either the HTTP client took care of this for me, or I had to add a filter to the web server using GZIPInputStream.

[Ron McLeod]

Here's an example of how to decompress with GZIPInputStream (I am using the built-in HTTP client).  If you cannot find a built-in capability with RestTemplate, maybe you could do something like this.

Note: this is proof-of-concept code - I didn't pay any attention to exceptions, closing resources, etc.

import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpClient.Redirect;
import java.net.http.HttpClient.Version;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse.BodyHandlers;
import java.time.Duration;
import java.util.regex.Pattern;
import java.util.zip.GZIPInputStream;

void downloadTest() throws IOException, InterruptedException {

    final var client = HttpClient.newBuilder()
            .version(Version.HTTP_1_1)
            .followRedirects(Redirect.NORMAL)
            .connectTimeout(Duration.ofSeconds(20))
            .build();

    final var request = HttpRequest.newBuilder()
            .GET()
            .uri(URI.create("http://api.planets.nu/games/list?username=wasp"))
            .build();

    var response = client.send(request, BodyHandlers.ofInputStream());
    boolean successful = Integer.toString(response.statusCode()).startsWith("2");
    if (!successful) {
        System.err.println("Unsuccessful .. stopping");
        return;
    }

    var contentEncoding = response.headers().firstValue("Content-Encoding").orElse("");
    var contentType = response.headers().firstValue("Content-Type").orElse("text/plain");
    var charSet = getCharSetFromContentType(contentType);

    var content = isGzipEncoded(contentEncoding)
            ? decompressGzipStreamToString(response.body(), charSet)
            : new String(response.body().readAllBytes(), charSet);

    System.out.println(content);
}

String decompressGzipStreamToString(InputStream compressedInputStream, String charSet) throws IOException {
    GZIPInputStream gzInputStream = new GZIPInputStream(compressedInputStream);
    var decompressedBytes = gzInputStream.readAllBytes();
    return new String(decompressedBytes, charSet);
}

boolean isGzipEncoded(String contentEncoding) {
    return contentEncoding.contains("gzip");
}

String getCharSetFromContentType(String contentType) {
    var matcher = Pattern.compile("charset=(?<charset>\\S+)").matcher(contentType);
    return matcher.find()
            ? matcher.group("charset")
            : "UTF-8"; // default
}

[Ricardo Carvalho]

Thank you very much.

That worked beautifully. Couldn't have done without you.

You guys rock.

Now i'll try to understand better the code you wrote. And try to find why mine didn't work.
I wish Java could simplify things . Other languages are much simpler.