How to decode response body?

[LCF-AT]

Hi guys,

so I found a new problem I din't had before yet and just wonder how to deal with that.I was trying to request diffrent servers on internet and somehow by random I get diffrent results back about the response body.

Case 1: Responses fine with status 200 & body after header is readable (some text)

Case 2: Responses fine with status 200 & body after header is NOT readable (encoded)

It dosent matter what kind of paramters I do send to the servers so the results is always randomly.I checked that also out in Fiddler by sending my request with WinSSL functions and there I get same issues (sometimes response is clean / sometimes is encoded).The question now is how to handle = what kind of request paramters I MUST send to prevent getting encoded datas from server (wanna get it clean readable as normaly) OR I need to know how to decode the body after I got it.So just to prevent that would be better of course.

First questions is what kind of encoding is used.In the response header I can see the info about...

Content-Encoding: gzip

....each time I get that response in case of encoded body.Ok, in that case its maybe gzip encoded for sure but why does send the server this to me and not clean?I didnt request any accept info to send me response encoded back.Thats the problem and it seems it and server bad config issue.I was looking to find any solution and somewhere I found a info to send...

Accept-Encoding: identity

...to the server and get all in text / clean / not encoded but also this dosent work. Seems that I have to handle it by myself after getting the response body and doing some gzip unpack.In Fiddler is a button to encode it but how to encode the gzip body with any xy function?I have here any zlib lib, which should work right but which function/s to use for memory block?Can anyone help with some example infos?Thanks.

greetz

[whoknows]

gzip = Deflate compression method

 

ref - dev.to/biellls/compression-clearing-the-confusion-on-zip-gzip-zlib-and-deflate-15g1

--

Accept-Encoding developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Accept-Encoding
Content-Encoding developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding

Edited June 3, 2021 by whoknows

[LCF-AT]

Hi,

ok but what kind of functions I have to use for it?Listen, I found some zlib on site from fearless.....

https://github.com/mrfearless/libraries/blob/master/Zlib/Zlib x86/zlib.inc

....now I need to know which functions I have to use to decompress the gzip compression from a buffer...so you said I  have to use deflate...

deflateinit()
deflate(Z_FINISH)
deflateEnd()

...now I made this...

zlibVersion             PROTO                                                           ; const char * ZEXPORT zlibVersion OF((void));
--
deflateInit             PROTO :DWORD,:DWORD                                             ; int ZEXPORT deflateInit OF((z_streamp strm, int level));
deflate                 PROTO :DWORD,:DWORD                                             ; int ZEXPORT deflate OF((z_streamp strm, int flush));
deflateEnd              PROTO :DWORD                                                    ; int ZEXPORT deflateEnd OF((z_streamp strm));
--
deflateInit_            PROTO :DWORD,:DWORD,:DWORD,:DWORD                               ; ZEXTERN int ZEXPORT deflateInit_ OF((z_streamp strm, int level, const char *version, int stream_size));

z_stream                        STRUCT
    next_in                     DWORD ?
    avail_in                    DWORD ?
    total_in                    DWORD ?
    next_out                    DWORD ?
    avail_out                   DWORD ?
    total_out                   DWORD ?
    msg                         DWORD ?
    state                       DWORD ?
    zalloc                      DWORD ?
    zfree                       DWORD ?
    opaque                      DWORD ?
    data_type                   DWORD ?
    adler                       DWORD ?
    reserved                    DWORD ?
z_stream                        ENDS


	invoke GlobalAlloc,GMEM_ZEROINIT+GMEM_FIXED,sizeof z_stream
	mov edi, eax
	lea eax, gzip
	mov [edi].z_stream.next_in,		eax  ; gzip buffer
	mov [edi].z_stream.avail_in,	        22h  ; lenght of gzip in buffer
	lea eax, BUFFER
	mov [edi].z_stream.next_out,	        eax  ; output buffer
	mov [edi].z_stream.avail_out,	sizeof  BUFFER
	
	invoke zlibVersion
	invoke deflateInit_,edi,NULL,eax,sizeof z_stream
	invoke deflate,edi,4      		     ; z_finish
	invoke deflateEnd,edi

....but dosent work.So on deflateInit_ I get 0 back = ZIP_OK.Next on deflate I get 1 back = Z_STREAM_END and next on deflateEnd I get  0 back = ZIP_OK.Anyhow I dont get the right stuff back.How to do it now?Any example I can follow correctly?

greetz

[kao]

20 hours ago, LCF-AT said:

I didnt request any accept info to send me response encoded back.

That's exactly according to specification. See https://datatracker.ietf.org/doc/html/rfc7231#section-5.3.4:

Quote

A request without an Accept-Encoding header field implies that the user agent has no preferences regarding content-codings.

 

"Accept-Encoding: identity" should have worked, even though the proper way to refuse gzip is to send Accept-Encoding with gzip and qvalue=0. Something like this:

Accept-Encoding: identity, gzip;q=0

 

5 minutes ago, LCF-AT said:

I dont get the right stuff back

You have lots of things backwards in your code.

* "deflate" compresses data. To decompress data you need to call "inflate" .
* you will probably need to call "inflateInit2" instead of normal inflateInit(). See https://stackoverflow.com/a/1838702
* it's a bad idea to set buffers before calling {whatever}Init() functions. In this case it will probably work, but quite often {whatever}Init() functions reset all fields in the structure, including buffer pointers and sizes.

There are plenty of example showing how to use ZLIB properly, so just google.

[LCF-AT]

Hi,

thanks for the infos.I tried this already out...

Accept-Encoding: identity, gzip;q=0

...but also didnt work because the server must be using a bad config or whatever.Sometimes it sends not compressed back and sometimes compressed anyway if I use this paramter above or not = sucks.Now I need to check the body manually and doing a gzip decompressing of the buffer but here I have problems again to find any example because all I can find is bad or manuals sucks to check something good what makes me sick again.

Ok I made this now....

	invoke GlobalAlloc,GMEM_ZEROINIT+GMEM_FIXED,sizeof z_stream
	mov edi, eax
	
	lea eax, gzip
	mov [edi].z_stream.next_in,		eax  ; gzip buffer
	mov [edi].z_stream.avail_in,	22h	 ; lenght of gzip in buffer
	lea eax, BUFFER
	mov [edi].z_stream.next_out,	eax  ; output buffer
	mov [edi].z_stream.avail_out,	sizeof BUFFER
	
	
	invoke zlibVersion
	invoke inflateInit2_,edi, 16+15,eax,sizeof z_stream  ; 16 + MAX_WBITS = 1Fh
	invoke inflate,edi,4  ; Z_FINISH
	invoke inflateEnd,edi

....now it seems to work. The buffer was decompressed and I can see my example 123 text in outbuffer. Oh men!Another hard birth.

PS: Ok I will set my buffers init function later.Just need to find out later how to use the other decompress styles of other compressions which can be send by response header I have possible to handle manually too.I am just glad that this now works for gzip at the moment.Thanks again for that info kao.

greetz