LCF-AT Posted December 14, 2017 Author Posted December 14, 2017 Hi, do you know any standard solutions I could use for that? Its pretty bad to find any detailed informations and examples what to do in case A,B,C etc.As I said,my main goal is it to handle only that redirection codes till I get status success or status error so that I can stop to request go on and going in any loop process without to finish.Lets say a site only allows to redirect to https and dosent send that info in location field then I trigger a loop.Found no site yet (except google which also allows http too).Maybe a little small steplist would be helpfully to know what to do and when to stop. Redirection messages -------------------------------------------------------------- | 300 | Multiple Choices | Section 6.4.1 | | 301 | Moved Permanently | Section 6.4.2 | | 302 | Found | Section 6.4.3 | | 303 | See Other | Section 6.4.4 | | 305 | Use Proxy | Section 6.4.5 | | 306 | (Unused) | Section 6.4.6 | | 307 | Temporary Redirect | Section 6.4.7 | 308 too -------------------------------------------------------------- Lets say I send first request via http port 80 and get any of this RD status codes back. Check location field if present and read it.Check if https:// is present at location field start = use https port 443 and also check if host is same or changed or used www or not and change it.If none of both are present = no port change / no host change & use relativ path found in location for next request.Something like that just to do a correct check and update to handle redirection. About HSTS.Does it mean its just checking a whole list to check whether server xy is found and if so then switch to HTTPS and I can not check the response header for that like for some other sites which do send https protocol at location field? greetz
LCF-AT Posted December 16, 2017 Author Posted December 16, 2017 Hi, I am looking for a API as fgets to read lines but not from any loaded file (from memory / buffer address xy).So it seems I cant use fgets for that.Now I tried also a around with scanf function (%[^\n]) but its not workiing. Its also possible to disable case sensitiv with scanf? https://%s Now it checks for https but if HTtps is in buffer = fail greetz
LCF-AT Posted February 19, 2019 Author Posted February 19, 2019 Hi guys, I was working on my internet stuff again and I have a question about using HTTP/1.1 version in my request header.I found some issues which I should handle extra but I would like to ask too about it. Problem: Normaly I call the function recv (to receive incomming bytes) so long till the return value in EAX is NULL (no more bytes comming) or -1h (SOCKET_ERROR).All working so far if I use HTTP/1.0 version.Now if I use instead HTTP/1.1 in my request and I do call my recv function loop then it does hang inside for a very long time before it comes back.I did check that code in OllyDBG to see what the reason for this behavior could be.I made a simple request.... GET / HTTP/1.1 Host: www.google.com ...and it does call recv function twice without any hang issues and at this point I got all bytes into buffer (Chunked) and on next call it hangs.My question now is whether I first have to check the header whether "Transfer-Encoding: chunked" is present and if YES = Do not call recv function till it does return 0 and just check the Chunks in buffer for end Chunk 0 / CRLF to know that I got already all bytes?What is if no "Transfer-Encoding: chunked" present,can I then call recv so long till it returns 0? I am also setting a timeout via ioctlsocket / select function.Not sure whether this could be the problem that it hangs inside recv function. Maybe anyone has some hints about that. One more question about that "Transfer-Encoding: chunked".On internet I found just some URLs using normal chunk method but I also read something about chunk extension using ; sign.Does anyone know any URL who is using this kind of chunk extension?Just wanna check and see that in real to know how to handle it. Thank you
LCF-AT Posted March 2, 2019 Author Posted March 2, 2019 Hi guys, I am still trying to write a code to handle that Transfer-Encoding chunked bytes but I am getting total confused each time with the entire checking issues and logging the rest chunk size / buffer address of not finished chunks! Maybe anyone of you can help with some hints how to handle that thing. My goal is it to handle / copy the raw chunk bytes in 2 diffrent ways.First I just wanna copy the raw bytes on fly (file / handle) and in a other way I just wanna keep and copy the raw bytes into my buffer I use already for recv function (max 10 MB only). I am looking for any advice / ideas how to handle it best so I need some input from others (you) how you would do that you know.I tried alredy to write a function for that.... HandleChunkedBlocks2 proc uses edi esi buf:DWORD,lenght:DWORD ,ZeroLenght:DWORD, KeepSource:DWORD local IsLenght:DWORD local SINGLECHUNKSTRINGLENGHT:DWORD local copiedbytes:DWORD local newbuf:DWORD push buf pop newbuf mov copiedbytes, 0 mov esi, buf mov ecx, lenght @@: .if ecx < 5 ; smaller 5 mov eax, TRUE ; loop recv ret .else .if byte ptr [esi] == ("0") && dword ptr [esi+1] == 0A0D0A0Dh ; last chunk present mov eax,FALSE ; Finish ret .elseif byte ptr [esi] == NULL ; NULL byte into / call recv again mov eax, TRUE ret .else mov IsLenght,ecx invoke crt_strtol,esi,NULL,10h .if eax != FALSE mov ebx, eax mov SINGLECHUNKSTRINGLENGHT,0 .while eax != FALSE inc SINGLECHUNKSTRINGLENGHT shr eax, 4 .endw mov eax, SINGLECHUNKSTRINGLENGHT .if word ptr [esi+eax] == 0A0Dh ; check for CRLF after value string add ebx, eax ; string add temporary .if ebx < IsLenght ; check if rest lenght of buffer is higher than chunk block itself .if word ptr [esi+ebx+2] == 0A0Dh ; chunk end CRLF / full single chunk present sub ebx, eax ; string sub temporary sub IsLenght, ebx ; sub chunk lenght mov eax, SINGLECHUNKSTRINGLENGHT sub IsLenght, eax ; sub string lenght sub IsLenght, 2 ; sub CRLF sub IsLenght, 2 ; end CRLF ; IsLenght = rest size ; ebx chunk size push ebx ; bak to stack lea edi,[esi+eax+2] ; raw chunk start to edi nop ; copy bytes edi / ebx to file etc nop ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; .if KeepSource == TRUE pushad push ebx add copiedbytes, ebx invoke copy,edi,newbuf,ebx,FALSE pop ebx add newbuf, ebx mov edi, newbuf mov word ptr [edi], 0 popad .endif ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; pop ebx ; restore chunk size to ebx mov eax,SINGLECHUNKSTRINGLENGHT add esi, eax ; add string lenght to esi add esi, 2 ; add CRLF size after string to esi add esi, ebx ; add raw chunk size to esi add esi, 2 ; add end CRLF lenght to esi = next chunk start mov ecx, IsLenght ; rest size of buffer to ecx jmp @B ; loop / check next chunk .else @@: sub ebx, eax ; string sub temporary ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; .if KeepSource == TRUE pushad mov ebx, IsLenght add copiedbytes, ebx invoke copy,esi,newbuf,ebx,TRUE popad mov ecx, copiedbytes mov eax,TRUE ret .endif ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; invoke copy,esi,buf,IsLenght,FALSE ; copy not finished chunk bytes to buffer top mov edi, buf ; buffer top to edi add edi, IsLenght ; add rest lenght to edi = end of not finished chunk mov ecx, ZeroLenght ; whole buffer size to ecx sub ecx, IsLenght ; sub rest size from whole buffer size xor eax, eax ; set eax to NULL REP STOS BYTE PTR [EDI] ; zero rest buffer mov ecx, IsLenght ; rest chunk size to ecx mov eax, TRUE ; eax TRUE / loop recv function again ret .endif .else jmp @B .endif .elseif byte ptr [esi+eax] == (";") ; is extension mov eax, -2h ; is extension / not supported yet ret .else ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; .if KeepSource == TRUE pushad invoke copy,esi,newbuf,IsLenght,TRUE mov eax, IsLenght add copiedbytes, eax popad mov ecx, copiedbytes mov eax, TRUE ret .endif ;;;;;;;;;;;;;;;;;;;;;;;;;;;;; invoke copy,esi,buf,IsLenght,FALSE ; copy not finished chunk bytes to buffer top mov edi, buf add edi, IsLenght mov ecx, ZeroLenght sub ecx, IsLenght xor eax, eax REP STOS BYTE PTR [EDI] mov ecx, IsLenght mov eax, TRUE ret .endif .else mov eax, -1h ; strol failed / aboard ret .endif .endif .endif Ret HandleChunkedBlocks2 endp ....but anyhow I still have problems to handle the rest sizes before calling my function... add eax, LASTCHUNK_REST_SIZE ; rest chunk size or NULL sub esi, LASTCHUNK_REST_SIZE ; sub rest chunk size or NULL from buffer VA in esi mov ebx, RecvSpaceLeft ; whole buffer size to ebx invoke HandleChunkedBlocks2,esi,eax,ebx,FALSE .if eax == TRUE nop mov LASTCHUNK_REST_SIZE,ecx add esi, ecx .elseif eax == FALSE .break ; finished .else ; error aboard .endif ...all in all it seems not so easy or its just to hard for me to get that in my head etc.Any ideas / hints are welcome also just some theory would be fine if you have something in your head for this etc. greetz
LCF-AT Posted March 3, 2019 Author Posted March 3, 2019 Hi again, questions: Are there some special cases XY I need / must use the HTTP/1.1 or higher version for any request I do?Can this also be rejected maybe so that I must use HTTP/1.1 or higher instead of HTTP/1.0?Just asking so I am getting totaly crazy with that trash chunked BS what is really PITA! No clue how other tools handle that chunks.I think they doing just any simple check/s without to check all possible issues which can happen and aboard then the process if any condition dosent match. Example: Lets say this is whole response and also imagine you get just 3 bytes size back after recv function Also dont care about the resonse header in this example. How to handle & loop now is the question. ----------------------------------------- 7\r\n Mozilla\r\n 9\r\n Developer\r\n 7\r\n Network\r\n 0\r\n\r\n ----------------------------------------- My Checkings I have to do - checking size after recv function whether its minimum 5 bytes long (0\r\n\r\n = 5 bytes of last chunk header) - reading value string of chunk lenght via strol function to get lenght in EAX of chunk - shr eax, 4 / inc ecx loop till eax NULL = stringlenght in ecx now - checking buffer + stringlenght for CRLF after string (stringvalue\r\n) - compare received size from recv function (minus stringlenght & CRLF after) with size of this chunk - checking for end CRLF of this chunk - adjust size & buffer If the checkings above do match then I can copy the raw chunk to file etc Now handle next chunk.... So on paper it looks anyhow pretty simple but its not (for me). Does anyone know any source XY what does handle chunks or has any of you any ideas which could help? greetz
LCF-AT Posted March 5, 2019 Author Posted March 5, 2019 Hi again, ok I think I got it working now. I just do copy the not finished chunk on the buffer top (overwrite) with the rest lenght and zero the rest.The chunks which are complete I do copy in fly to file / hande OR new buffer (keep bytes with a size limit).My function looks so now. ; download chunks to file / handle pipe ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; mov ecx, LASTCHUNK_REST_SIZE sub esi, ecx ; esi = static buffer VA add ecx, READSIZE ; Readsize = from recv + LASTCHUNK_REST_SIZE invoke CopyChunksOnFly,esi,ecx,FIRSTADDRESS,RecvSpaceLeft,NULL .if eax == TRUE mov LASTCHUNK_REST_SIZE, ecx ; edx = raw chunksize we can add to size of download add esi, ecx ; ecx = rest chunk size which isnt handled .continue ; call recv again .elseif eax == FALSE ; finished .break .else ; Error -1 or -2 .endif ; copy raw chunks to new buffer ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; push eax ; size from recv function mov eax, MAX_RECV_BUFFER_SIZE ; 10 MB mov ecx, 2 ; div 2 / 5 MB limit xor edx,edx div ecx mov ecx, eax pop eax .if ecx < MYHEAPKEEP_TEMP ; MYHEAPKEEP_TEMP = raw chunksizes we have already ; when MYHEAPKEEP_TEMP higher 5 MB = stop .endif mov ecx, LASTCHUNK_REST_SIZE sub esi, ecx add ecx, READSIZE mov ebx, MYHEAPKEEP ; new buffer add ebx, MYHEAPKEEP_TEMP ; adding chunk raw sizes invoke CopyChunksOnFly,esi,ecx,FIRSTADDRESS,RecvSpaceLeft,ebx .if eax == TRUE mov LASTCHUNK_REST_SIZE, ecx ; edx = raw chunksize we can add to size of download add esi, ecx add MYHEAPKEEP_TEMP, edx .continue .elseif eax == FALSE ; finished ; check for something in source buffer of MYHEAPKEEP .break .else jmp @ERROR .endif CopyChunksOnFly proc uses edi esi buf:DWORD,lenght:DWORD,Top:DWORD,FullBufferLenght:DWORD,NewBuf:DWORD local SINGLECHUNKSTRINGLENGHT:DWORD local CHUNKSIZE:DWORD local IsLenght:DWORD local CHUNKSIZEADDED:DWORD local IsNewBuf:DWORD mov ecx,lenght mov IsLenght,ecx mov esi, buf mov CHUNKSIZEADDED, 0 push NewBuf pop IsNewBuf @@: .if ecx < 5 invoke copy,esi,Top,IsLenght,FALSE mov edi, Top add edi, IsLenght mov ecx, FullBufferLenght sub ecx, IsLenght ; rest lenght to zero xor eax, eax REP STOS BYTE PTR [EDI] ; zero rest buffer mov ecx, IsLenght mov eax, TRUE .else .if byte ptr [esi] == ("0") && dword ptr [esi+1] == 0A0D0A0Dh ; last chunk present mov eax, FALSE ; successfully found last chunk body / finsih .else invoke crt_strtol,esi,NULL,10h .if eax != FALSE mov ebx,eax mov CHUNKSIZE, eax mov SINGLECHUNKSTRINGLENGHT,0 .while eax != FALSE inc SINGLECHUNKSTRINGLENGHT shr eax,4 .endw mov eax, SINGLECHUNKSTRINGLENGHT .if word ptr [esi+eax] == 0A0Dh ; check for CRLF after value string sub IsLenght, eax sub IsLenght, 2 .if ebx <= IsLenght add ebx, eax ; string add temporary .if word ptr [esi+ebx+2] == 0A0Dh ; chunk end CRLF / full single chunk present sub ebx, eax ; string sub temporary add CHUNKSIZEADDED, ebx sub IsLenght, 2 sub IsLenght, ebx lea edi,[esi+eax+2] ; raw chunk start to edi push ebx nop nop .if NewBuf != FALSE push ebx invoke copy,edi,IsNewBuf,ebx,FALSE ; copy to new buffer pop ebx add IsNewBuf, ebx ; add chunk size to new buffer .else nop ; copy bytes edi / ebx to file etc .endif nop nop nop pop ebx mov eax,SINGLECHUNKSTRINGLENGHT add esi, eax ; add string lenght to esi add esi, 2 ; add CRLF size after string to esi add esi, ebx ; add raw chunk size to esi add esi, 2 ; add end CRLF lenght to esi = next chunk start mov ecx, IsLenght ; rest size of buffer to ecx invoke copy,esi,Top,IsLenght,FALSE mov edi, Top add edi, IsLenght mov ecx, FullBufferLenght sub ecx, IsLenght ; rest lenght to zero xor eax, eax REP STOS BYTE PTR [EDI] ; zero rest buffer mov ecx, IsLenght mov esi, Top jmp @B .else @@: add IsLenght, eax add IsLenght, 2 sub ebx, eax ; string sub temporary invoke copy,esi,Top,IsLenght,FALSE mov edi, Top add edi, IsLenght mov ecx, FullBufferLenght sub ecx, IsLenght ; rest lenght to zero xor eax, eax REP STOS BYTE PTR [EDI] ; zero rest buffer mov ecx, IsLenght mov eax, TRUE .endif .else add ebx, eax ; string add temporary jmp @B .endif ;;;;;;;;;;;;;;;;;;;;;;;; .elseif byte ptr [esi+eax] == (";") ; is extension mov eax, -2h .else invoke copy,esi,Top,IsLenght,FALSE ; copy not finished chunk bytes to buffer top mov edi, Top add edi, IsLenght mov ecx, FullBufferLenght sub ecx, IsLenght ; rest lenght to zero xor eax, eax REP STOS BYTE PTR [EDI] ; zero rest buffer mov ecx, IsLenght mov eax, TRUE .endif .else mov eax, -1h ; strol failed / aboard .endif .endif .endif mov edx, CHUNKSIZEADDED ; retrun full raw chunk bytes which was handled Ret CopyChunksOnFly endp I thinks its right now to be sure to handle all normal chunks right (I hope so). greetz
LCF-AT Posted March 16, 2019 Author Posted March 16, 2019 Hi again, short question.I forgot how to compare 2 QWORDs with each other to check whether both are same or not etc.Can anyone refresh the commands for that for me?Sorry I dont find the code for that at the moment in my notices anymore.Something with FLD / Float commands etc.Found and tested something like this... FINIT FLD QWORD PTR [CONTENTLENGHTSIZE] FLD QWORD PTR [RECVSIZE] FCOMI ST,ST(1) je @EQUAL ; jump if equal = stop calling recv function ja @EQUAL ; jump if RECVSIZE higher = same ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; in Olly 00823D1C WAIT 00823D1D FINIT 00823D1F FLD QWORD PTR SS:[EBP-0x9C] 00823D25 FLD QWORD PTR SS:[EBP-0x94] 00823D2B FCOMI ST,ST(1) 00823D2D JE SHORT 00823D38 00823D2F JA SHORT 00823D38 ...seems to work on testing a little bit.Just wanna know it for sure whether I can keep this code like this.I have seen that in some cases I need to check & compare the received raw size with the content-lenght from header (using HTTP/1.1) if present (also if no chunked transfer is used) to prevent hanging in recv function.Not happens if I use HTTP/1.0. greetz
CodeExplorer Posted March 17, 2019 Posted March 17, 2019 (edited) https://stackoverflow.com/questions/7057501/x86-assembler-floating-point-compare https://c9x.me/x86/html/file_module_x86_id_88.html Quote ja @EQUAL ; jump if RECVSIZE higher = same This part looks weird to me, why if is bigger will jump to @EQUAL ??? Edited March 17, 2019 by CodeExplorer 1
LCF-AT Posted March 17, 2019 Author Posted March 17, 2019 Hi, thanks for the links.I think my code above seems to be ok for my task to make a compare of both local QWORDs.The "ja" command I did just set to jump also to equal label if this added size is higher inside.Just to be sure so normaly its not getting higher of course.Just wanna jump if equal or higher (should not happens) etc you know.So in both cases I have to quit and leave the byte receiving loop. greetz
LCF-AT Posted November 4, 2019 Author Posted November 4, 2019 Hi guys, after long time of checking my internet stuff I still have a little issue to stuck in a function for a long time also with setting a timeout. Problem 1: I do stuck a longer time in the select function before it returns = Why? Problem 2: I do stuck a longer time in the SSL_write function before it returns = Why?Also if I send less bytes.I thought it would also aboard if the timelimit has reached but no. Those problems happens anyhow randomly more or less.I wrote a realtime logger before accessing any function and to see where it stucks for a longer time and mostly its just the select & SSL_write function.My question now are how to prevent this stuck process in those function and to force a aboard?One time it did stuck on SSL_write forever without to return anymore and I had to quit my app.Just wanna prevent such problems if possible.I am using the ioctlsocket & ioctlsocket & select method to set a timeout on write postet before by evlncrn8.Below a example code I am using similar (MASM style). bool connect(char *host,int port, int timeout) { TIMEVAL Timeout; Timeout.tv_sec = timeout; Timeout.tv_usec = 0; struct sockaddr_in address; /* the libc network address data structure */ sock = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP); address.sin_addr.s_addr = inet_addr(host); /* assign the address */ address.sin_port = htons(port); /* translate int2port num */ address.sin_family = AF_INET; //set the socket in non-blocking unsigned long iMode = 1; int iResult = ioctlsocket(sock, FIONBIO, &iMode); if (iResult != NO_ERROR) { printf("ioctlsocket failed with error: %ld\n", iResult); } if(connect(sock,(struct sockaddr *)&address,sizeof(address))==false) { return false; } // restart the socket mode iMode = 0; iResult = ioctlsocket(sock, FIONBIO, &iMode); if (iResult != NO_ERROR) { printf("ioctlsocket failed with error: %ld\n", iResult); } fd_set Write, Err; FD_ZERO(&Write); FD_ZERO(&Err); FD_SET(sock, &Write); FD_SET(sock, &Err); // check if the socket is ready select(0,NULL,&Write,&Err,&Timeout); if(FD_ISSET(sock, &Write)) <----- How is this check working? { return true; } return false; } In my case I do check for eax = 0 what means timeout expired,and checking for SOCKET_ERROR (-1).When the return in eax after calling select function is else = success.In my case its 1 for the count of the socket so I am just using one only.Ok so far.In the code above comes a check with the macro FD_ISSET on write fd_set struct.So what does it check there?Does it check whether the socket handle is still in this struct present I did moved before into?Or does it check the count value etc?Not sure about that yet.I tried to produce a manually error in select function to see what happens in this struct but nothing happens there and the values I did set before in this write struct are still same and wasnt zero-d or something.I also tried to fill the Err fd_set struct with count 1 & same socket handle and in this only the count value gets zero-d.Maybe anyone can tell me how to check this correctly in ASM / MASM or just telling simple.I just wanna produce a code with timeout (always using it) which works for 100% without to hang anywhere for a long time or forever you know. AddOn question: So if I see it right then I can set a timeout for send / SSL_write = write fd_set struct & recv / SSL_read = read fd_set struct and this Error fd_set struct.What about a connection timeout for connect function?Also doable or needed?Just asking.Would be nice if anyone could help a little with that whole timeout issues and how to make it correctly for 100% to prevent hangs anywhere. Thank you
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now