How to access the internet?

[simple]

Easiest for gzip is zlib (search for "gzip zlib masm"). I use it in my http libs. Free, open source.

1st thing is to parse that header -every value- according to http rfc's. If u r writing ur own parser for the header make sure to get it right, if not it will screw up the next step. Its a lot of code.

In ur response, the server provides no content-length header, look in rfc2616 (https://www.ietf.org/rfc/rfc2616.txt), the section on "content-length" header is on page 119. It details how to parse a message if content/transfer/message-length etc is not given, by referring to section 4.4 pg 33 & 34.  Pg 33, section 4.4 item #3 is your solution. Which then refers to specifications for chunked transfer, found on pg. 25 section 3.6.1.

So u can see your chunk size in the header, I assume u r using normal sockets & not SSL sockets, so in your recv() call u want to make sure u get that many bytes, until transfers complete, then decompress to get the content.

[LCF-AT]

Hi and thanks so far,

so do you mean now I have not to send another request with new added paramters or do you mean I have to decompress the encoded bytes anyhow with zlib like if I would unzip any zip file?

Only can find this below which has any zlibstat128.lib etc but not sure whether its right or not what you mean.

http://masm32.com/board/index.php?topic=421.15

Yes its not SSL site I do request and just wanted to know how to decode the compressed / chunked / Encoded code I got back.

greetz

[simple]

No, read the 1st chunk size, take that value, make sure u read that # of bytes from recv(). Once u hit that # of bytes, extract them, do whatever, then parse your next chunk size, then repeat, until you reach a chunk size of 0 or server terminates connection. Just keep staring at the RFC's and reading server responses, ull get it.

[LCF-AT]

Hi,

can you also explain it more simple simple?How to decode the bytes?Fiddler dosen't make another request if I press the button to decode it so I did monitor the internet accessing and there happend nothing what means that Fiddler does do it by itself with any decode operation.

This is what I got...

$ ==>    00DE0170   32 61 35 66 0D 0A 1F 8B 08 00 00 00 00 00 00 03  2a5f..‹......
$+10     00DE0180   ED 7D 7B 7B DA B8 F2 F0 DF C9 A7 D0 F1 76 0B 9C  í}{{Ú¸òðßɧÐñvœ
$+20     00DE0190   70 33 84 24 90 40 9F 5C 7A DB 6D DA BC 4D D2 9E  p3„$@Ÿ\zÛmÚ¼MÒž
$+30     00DE01A0   F3 74 FB F2 08 5B 80 13 63 BB BE 40 D8 B6 DF FD  ótûò[€c»¾@ضßý
$+40     00DE01B0   37 23 C9 37 EE E9 92 6C CE 36 79 5A B0 E5 91 34  7#É7îé’lÎ6yZ°å‘4
$+50     00DE01C0   9A 9B 66 24 A3 39 F8 D7 C9 BB E3 8B FF 9E 3D 27  š›f$£9ø×É»ã‹ÿž='
$+60     00DE01D0   7D 7F 60 B6 36 0F F0 8B 98 D4 EA 35 89 C2 2C 85  }`¶6ð‹˜Ôê5‰Â,…

$+2A45   00DE2BB5   78 19 5C B5 C6 35 78 DE 46 2A 87 56 A9 63 EB 63  x\µÆ5xÞF*‡V©cëc
$+2A55   00DE2BC5   64 16 CE 86 AD CD FF 03 54 04 E3 D9 F1 E7 00 00  dΆ­ÍÿTãÙñç..
$+2A65   00DE2BD5   0D 0A 30 0D 0A 0D 0A 00 00 00 00 00 00 00 00 00  ..0.............

2a5f = lenght of chunk | \r\nSTART at 00DE0176 VA + 2a5f = 00DE2BD5 VA END CLRF

...what now?

PS: Simple solutions are welcome.

greetz

[simple]

I know RFCs are a pain in the @ss to read but they answer most of life's questions. Unless u want to purchase ur own http libs u have to get used to them.

RFC's on gzip are rfc1951 and rfc 1952

Industry standard for gzip is zlib, its one of the most well documented, open source libs I know of w/millions of code samples. I did not personally verify this but it looks right, if not search in C

http://www.masmforum.com/board/index.php?PHPSESSID=786dd40408172108b65a5a36b09c88c0&topic=9039.0

[LCF-AT]

Hi,

ok listen,I am just looking for a simple way / method to decompress the compressed bytes I got in memory also into memory without to dump the bytes on HDD etc.Now I just make a test and did dumped my exact bytes from start to end with right byte lenght to HDD and now I just reanmed the file extension to .rar and I can load & unrar this file now with winrar and I get the decompressed datas back and can read the page source text.Now I wanna know what dll / lib I need to use + the right APIs / Procs using it with my compressed datas in memory at VA start - lenght + new memory destination.Like...

invoke Decompress,MemStart,Lenght,NewMemDestination

...or maybe I could also use directly dll APIs (would be also ok) if there is any who can do it in memory.I also downloaded a dll from here..

http://zlib.net/zlib128-dll.zip

...not sure whether its right or not but it also has tons of APIs and I don't which I have to use etc.Maybe you can tell me what I could do in that case.Thanks.

greetz

[simple]

All the answers are in the RFC, zlib docs, etc etc. there's millions of code samples out there for this. I think you should consider buying a http lib. Anyways good luck.

[LCF-AT]

Ok,so can you tell what for ready usable libs are out there which I could use to prevent / solve request automatically?Maybe something like from

http://curl.haxx.se

...but I think we talked about this already right?There I also see many examples but also all in C only.

greetz

[fd8g7623]

hi, here is http request i wrote with winsocks, it's what you wanted?

code:

Spoiler

#include <iostream>

#include <winsock2.h>
#pragma comment(lib,"ws2_32.lib")

#include <string>

std::string get_source()
{
	WSADATA wsa;
	WSAStartup(MAKEWORD(2, 0), &wsa);

	SOCKET sock;
	SOCKADDR_IN sin;

	char csource[8192] = { 0 };
	char crequest[8192] = { 0 };

	std::string srequest = "GET ";
	srequest += " HTTP/1.1\r\n";
	srequest += "Host: google.com\r\n";
	srequest += "Connection: close\r\n";
	srequest += "\r\n\r\n";

	size_t request_size = srequest.size() + 1;
	strncpy(crequest, srequest.c_str(), request_size);

	sock = socket(AF_INET, SOCK_STREAM, 0);
	sin.sin_addr.s_addr = inet_addr("173.194.123.73");
	sin.sin_family = AF_INET;
	sin.sin_port = htons(80);

	connect(sock, (SOCKADDR *)&sin, sizeof(sin));
	send(sock, crequest, strlen(crequest), 0);

	int i = 0;
	std::string source;

	do
	{
		i = recv(sock, csource, sizeof(csource), 0);
		source += csource;
	} while (i != 0);


	closesocket(sock);
	WSACleanup();

	return source;
}

void main()
{
	std::cout << get_source() << std::endl << std::endl;

	system("PAUSE");
}

 

 

[LCF-AT]

Hi,

so I build already my request codes but thanks for posting.

So I have a another question about how to work with the received Header Datas so how to do it correctly?Lets say I do send a simple request with HEAD only to get the Header datas into my buffer.Now I have it all as ASCII string.Now I wanna just check the status value in the header which is maybe for example "302 Found".The question I have is how to check / parse it right.Now I wanna check the status for 200 then I only can use string APIs like strstr to check for "200 OK" for example in the received Header but this seems to be a little unsafe / strange.So on that way I have to use a lot of strings to check different paramters.So are there also other ways to check datas of the header like the WinInet API HttpQueryInfo can do where I can set paramters and check them (HTTP_QUERY_STATUS_CODE & HTTP_STATUS_OK) etc?Other problem is how to get the values in Hex like for "Content-Length: XY" to work with it (also if lenght is higher than 4 GB) etc you know.Seems that I have to do it all manually to check for xy strings and reading the dec values / convert them etc.Maybe anyone does know any better and simple ways how to handle and interpret the received Header datas.Thanks.

greetz

[LCF-AT]

Hi guys,

yesterday I found a new problem if I try to use WSock to send requests and I don't know how to handle this problem.The problem I got is a error after calling the API gethostbyaddr and as return I get in my error register the error "WSANO_DATA (00002AFC) = 11004".Problem seems to be that the API can't resolve the host IP address.On net I found this description...

http://www.sockets.com/err_lst1.htm#WSANO_DATA
http://www.sockets.com/err_lst1.htm#WSAHOST_NOT_FOUND

...but still don't know how to fix it if I get this problem.Just a info:Yesterday I got this problem for some links I wanted to check by WSock (all working fine with same links by WinInet) and today I check links again and now its working.So what should I do if I get the same errors again next time?Was the problem on my system etc and if yes why etc?

greetz

[LCF-AT]

Hi again,

I have a new question.Could anybody tell me what to do if I wanna determine a download speed during a download?I did determine the fullsize lenght and also the current lenght (in loop).All in all I wanna create something similar about download infos like in a browser where you can see the infos too...

Currentsize: X
Fullsize: X 
DL-Speed: X (1.2 MB/s)*
remaining time: X (00:02:43)*

* = I need to know how to get them

...maybe you know how to do or calc this to get these infos too.Thanks

greetz

[NOP]

For download speed you need to get elapsed Milliseconds and number of bytes downloaded and then to get Bytes per second...

  BytesPerSec = BytesDownloaded * 1000 / ElapsedMS

From there you can calculate kb/s or mb/s and time remaining easily based off that and the file size

Edited January 8, 2016 by NOP

[LCF-AT]

Ok thanks so far.I wrote a example for this part now but its again a little strange (much code etc) but it seems to work.

mov eax, 10485760  ; Bytes = 10 MB
mov ecx, 2300      ; 2,3 Seconds
nop
mov BYDW, 1000     ; x 1000
add dword ptr [MILLISECS], ecx
adc dword ptr [MILLISECS+4], 0
add dword ptr [CurrentSize], eax
adc dword ptr [CurrentSize+4], 0
fild qword ptr [CurrentSize]
fild BYDW
fmul
fild qword ptr [MILLISECS]
fdiv
nop
mov MBDW, 1024 
fild MBDW
fdiv           ; div 1 bytes to KB = 1164 KB Hex / 4452 KB Dec
fistp qword ptr [RESULTS_KB] ; KiloBytes
fild qword ptr [RESULTS_KB]  ; KiloBytes to ST
fild MBDW
fdiv           ; div 1 KB to MB    = 4 MB
fistp RESULTS  ; MB in RESULTS
;-----------------------------
mov eax, dword ptr RESULTS  ; MB to eax
mov edi, eax
mov ecx, 1024
xor edx,edx
mul ecx
mov ecx, dword ptr RESULTS_KB
sub ecx, eax
invoke wsprintf,addr timeText,chr$("%d.%03d MB/s"),edi,ecx
; 4.356 MB/s

Maybe there is any better and easier way (I don't know it of course).

greetz

[ragdog]

Quote

I wrote a example

You wrote it ??

[LCF-AT]

Sure after working with the examples from before + calc infos of NOP I got it finaly working now. Maybe my code isn't perfect so far but seems to be ok during my tests.Below a small picture.

greetz

[ragdog]

Looks good

Edited January 10, 2016 by ragdog

[LCF-AT]

Hi again,

just wanna ask whether anyone can tell me how to handle a request / response scenario for FTP.So it seems to work else as for HTTP and does use specific commands and ports (20&21).Just wanna know how the syntax has to look which I must send (similar as for HTTP requests GET /* HTTP/1.0 etc) only for downloading a file.Thanks.

greetz

[simple]

Unless you want to forward ports on your router PASV mode RETR will be your option. U will have to do some string parsing, and u wont be able to use strstr() for it like on http headers heh.

Unlike HTTP the implementation of rfc959 FTP can be quite different from sever to server. 

Again, this is different from server to server but generally you login, set file type, set PASV mode, connect back to PASV port, then RETR filename.bin then listen on data port. U may have to set security type, enumerate the remote dir first, among other things. FTP is easy though, so that's the good news.

[LCF-AT]

Hi guys,

have again a little problem and need some help.This time I have a problem with the InternetOpenUrl API.Normaly all is working fine with that but for some sites I try to check I get error message 12007 (ERROR_SXS_KEY_NOT_FOUND (00002EE7)).The question is why and why only for some sites.The error said it can't relsove the host.Now I checked a little manually and found also this.Lets say I do use gethostbyname API with the homepagename like www.google.com then it works or also if I use just goole.com its working too but for some sites I have test it only works if I NOT use www. before and here we got the problem with using InternetOpenUrl.Anyway whether I use http://hompage.com or http://www.homepage.com its not working (for some sites) and WinInet failed in that case.Now I tried to use WinSock APIs with same site and it gets same trouble but only if I use www. at gethostbyname API if I don't use www. then it works again.Now,later on send API I HAVE to use a host paramter using www. to get it work.Its a little bit strange but so its working.

All in all I wanna get WinInet working again but how?If I use www. in the URL then WinInet will call also send API also with www. in the Host paramter = failed.If I don't use www. in the URL then it will not access the send API and comes back with same error message.Thats really strange / bad.So how to handle this problem now?

Thanks

[LCF-AT]

Hi guys,

short question.So could anyone give some smart infos how to parse a http response?So I use WinSock and there isn't any API or else I could use and it seems I have to write all checks manually and there comes the problem.Just simple example.If I request any site xy and got received the header datas into buffer then I check first the status and this I only do so...

invoke crt_strstr,offset HEADERBUFFER,chr$("200 OK")

...and if found Ok but if not then I need to add more & more such bad checking commands and this also is bad.Do you have any better solution how to handle that stuff on any smarter way or else etc?

greetz

[LCF-AT]

Hi guys,

have a new question.During my tests with WinInet and WinSock I found another new problem between both.The problem this time I see is that I get 2 diffrent  results back into the response calling with the same request.So it seems that WinInet dosent  return the Set-Cookie paramters into response but why?So I get the Set-Cookie paramters using WinSock.Is it again any WinInet issue thats not handable or have I to use other WinInet API parameters or is it just not possible to get and  work with cookies using WinInet?

greetz

[fearless]

did you get he zlib stuff to work, just noticed the link above to some stuff i done on that on the masm32 forum.

[LCF-AT]

Hi,

so I did created a zlib1.lib from a zlib1.dll I found on my HDD and used the uncompress export...

invoke uncompress,DEST,addr DESTLEN,SOURCE,addr SOURCELEN

....so on that way I got it working to uncompress the file in  memory.

greetz

[evlncrn8]

grab the zlib source from the home site, and compile it, if i remember right you can make a static lib from it.. its in one of the compile options... much better than dll hell, i did it in pid, but for the life of me i cant remember how, i'll check later on thought if you need me to