How to access the internet?

[LCF-AT]

Hi guys,

 

can anyone of you tell me how to access any internet site via APIs xy?So I don't know how to explain it correctly but what I am looking for is very simple.So what I wanna do is to access any site link xy and reading the source code into memory.All this I wanna do with a extern tool / code without to use any browser etc.The only thing what I need to know is...

 

1.) What APIs can I use to call internetsites / specific links

2.) How to read source code of the link xy into memory

 

So I imagine like this....

push ascii string // http://siteABC.com/12345.html

call API xy
or with login datas
push ascii string // http://name:pass@siteABC.com/12345.html

call API xy
check whether link request & login was successfully

je ReadSource

jmp exit

///////////////////////

ReadSource:

push site handle

call ReadMemory // source code in memory x

.......

.......

.......

///////////////////////

End:

push logout

call exit

...something like this I mean so you know right.Also the same I wanna do with sites where I need to login with my login datas without to confirm any messagebox which you normaly get in browser etc.So the question is how I could do this?Maybe you could help again.

 

Thank you

[kao]

1) InternetOpen + InternetOpenUrl + InternetReadFile. Search for it - there are plenty of http downloader sources in ASM (hey, most malware downloads some parts from the net! )

2) InternetOpen + InternetConnect + HttpOpenRequest + HttpSendRequest + InternetReadFile. Yes, it's slightly more complicated, but InternetConnect allows you to specify user/pass.

[White]

Hi,I got a way in my app,but I am not sure it works in your situation.

 

So 

 

 

2.) How to read source code of the link xy into memory

 

Maybe it looks like these

	HINTERNET hNet = ::InternetOpen("Internet Access",PRE_CONFIG_INTERNET_ACCESS,NULL,INTERNET_INVALID_PORT_NUMBER,0);

	if(NULL == hNet)

	{

		return FALSE;

	}

	HINTERNET hUrlFile = ::InternetOpenUrl(hNet,"Your Site string",NULL,0,INTERNET_FLAG_TRANSFER_BINARY|INTERNET_FLAG_NO_CACHE_WRITE|INTERNET_FLAG_RELOAD,0) ;

	if(NULL == hUrlFile)

	{

		InternetCloseHandle(hNet);  

		return FALSE;

	}

	BOOL	bRead	=FALSE;

	DWORD	dwSize	=0,dwDownLoaded,dwTotalWrite	=0,dwError	=0;

	bRead	=InternetQueryDataAvailable(hUrlFile,&dwSize,0,0);

	if (bRead)

	{

		char	*pszBuffer = (char *)calloc(dwSize+0x1,sizeof(char)); // alloc memory for read

		bRead = ::InternetReadFile(hUrlFile,pszBuffer,dwSize,&dwDownLoaded);

		HANDLE hFile = CreateFileA("A file to write", GENERIC_WRITE, 

			FILE_SHARE_WRITE, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, 

			NULL); //Create Read file or using string struct to cache pszBuffer
		if (hFile !=INVALID_HANDLE_VALUE)

		{

			while (dwDownLoaded !=0)

			{

				pszBuffer[dwDownLoaded]='\0';

				dwTotalWrite	+=dwDownLoaded;

				bRead	=WriteFile(hFile,pszBuffer,dwDownLoaded,&dwDownLoaded,NULL);

				memset(pszBuffer,0,dwSize);

				InternetReadFile(hUrlFile,pszBuffer,dwSize,&dwDownLoaded);

			}

			dwError	=GetLastError();

		}

		free(pszBuffer);

		CloseHandle(hFile);

	}

	InternetCloseHandle(hUrlFile);

	InternetCloseHandle(hNet);

Hope it can help you a little.

[Encrypto]

There is also a method using winsock, socket, recv, send etc... 

Using this method however, you have to write the headers and all that jazz yourself. The higher level functions of wininet do that for you though.

[LCF-AT]

Hello kao & White,

 

thanks for your informations about what APIs I need to check & exsample. So now I need to study this API description etc so I never used any internet API before.

 

Question: So I see I get already some problems with the API parameter NAMES XY as "PRE_CONFIG_INTERNET_ACCESS" etc so I can't push these as name in Olly so for these paramater I need the push values as 100 for exsample.How can I handle this problem without to use WinASM itself? Also I didn't work a long time no more with WinASM but if I remember right then I need to add a wininet.lib for right?Hmmmm!

 

Ok listen.Do you have or know any exsample exe files already which I could check?Or maybe you could create any simple one how does access any site (google etc) and does read the source.Just only a question of course before I test endless to find the right values xy for some parameter names you know.

 

EDIT: Good ok Encrypto but also I have no ideas about winsock etc.VB stuff right?So all in all I am just looking for any simple solution to access any internet site + reading source code of this site into memory and thats all already.So I don't wanna do it manually and thats the reason why I am looking for a nice & lazy way.

 

Thank you

Edited October 30, 2014 by LCF-AT

[h4sh3m]

Hi dear LCF-AT

 

this api might help you in this case : URLDownloadToFile

 

 

 

Good Luck,

h4sh3m

[metr0]

LCF, the appropriate values are defined within the header files (wininet.h and similar), just include them in WinAsm. And yes, you'll also have to link the library. That's as lazy as it gets.

[LCF-AT]

Hi again,

 

ok I got it now a little bit to create a project in WinASM.All working so far as I imagine at the moment but I have one problem.So I also use the InternetQueryDataAvailable API but there I get not the right lenght value back!Just get a smaller size back and not the full size of the source.....why?

Invoke InternetQueryDataAvailable, FileHandle,addr Lenght,0,0

Have I to use any other parameter values at the end?So I wanna read the entire page source not just some bytes xy.Maybe I have done something wrong with the API before InternetOpenA & InternetOpenUrlA but also I see that I have not all parameter NAMES in my WinASM as postet by the exsample of White...so this below I don't have....

INTERNET_FLAG_TRANSFER_BINARY|INTERNET_FLAG_NO_CACHE_WRITE|INTERNET_FLAG_RELOAD

....so at the moment I can just read the entire source blind (alloc 100000 bytes in the hope that the source is not larger).Of course its a bad solution for this but maybe you know what the problem is in my case above.

 

greetz

[White]

Hi, sorry for a later resp.

 

1.

Header Declared in Wininet.h.

Library Use Wininet.lib.

DLL Requires Wininet.dll.

2.

BOOL InternetQueryDataAvailable(

  __in   HINTERNET hFile,

  __out  LPDWORD lpdwNumberOfBytesAvailable,

  __in   DWORD dwFlags,

  __in   DWORD_PTR dwContext

);

And MSDN Remarks,

This function returns the number of bytes of data that are available to be read immediately by a subsequent call to InternetReadFile. If there is currently no data available and the end of the file has not been reached, the request waits until data becomes available. The amount of data remaining will not be recalculated until all available data indicated by the call to InternetQueryDataAvailable is read.

3. For now,do you need any example ?

 

 

EDIT:

 

 

not the full size of the source

 

If you have a file and size's 1GB, do you have a free memory to cache that big file ?  

So I choose a loop to read the available size till its end.

Edited October 31, 2014 by White、、

[LCF-AT]

Hi again,

 

ok but how to make a loop with InternetQueryDataAvailable API to get the entire size of the source of any site?So I have test a little bit with different sites I for some sites I get the full source size back and for other sites not also if they have a smaller source size so thats a little bit strange.Thats the reason why I now use a reading size of 100000h bytes to be sure to get the full source size without to use the InternetQueryDataAvailable API.

invoke VirtualAlloc,0,100000h,MEM_COMMIT,PAGE_EXECUTE_READWRITE

....

Invoke InternetReadFile,FileHandle,SECTION,100000h,addr BytesRead

....

So yesterday I made got already some success with my test app to read the source / find strings xy + export them into log file.

 

So now I have a another question about searching text strings into source.So how can I do this in the best and simplest way?Just imagine I have load the full source in memory and now I wanna check for some ASCII strings inside and to log the data which comes right after the strings.

 

Exsample: I wanna find the string ABCDEFGH.So at the moment I just use a pattern dword compare of ABCD = 44434241 & 48474645

cmp dword ptr [edi], 44434241 ; ABCD
cmp dword ptr [edi+4], 48474645 ; EFGH

But this is not so good so I dont wanna always check for opcodes.So I wanna do directly with the strings to find them but how and what API can I use for this?Lets say I have a text source size of 1000h bytes and somewhere inside are these strings to find and now I wanna check this source only one time to find all of them without to know where the strings starts.Normaly I could use commands as "REPE CMPS BYTE PTR ES:[EDI],BYTE PTR DS:[ESI]" etc but for this I need first to find the string start to compare it then but this method I don't like.

 

So I am looking for a API like this..

push addr text start

push size of full text lenght to check = 1000 bytes for exsample

push string to find pointer = ASCII "ABCD"

call API_XY_Who_Does_Search_For_It
return = 0 = String not found in text

or

return = VA in eax where string starts of first found

ecx = rest size to check so that I can re-call the API
Something like this you know

...but I dont remember any API who could to this etc.So at the other hand I could use the API CompareString but then I have to call this API for each byte (no idea whether this is good or bad because performance speed etc).

 

Ok so I think there is no big different if I use opcode check as strings or compare APIs about the speed.So now I use this compare API.

 

PS: I also see that I have no API parameter to choose for the CompareString API in WInASM. So I cant choose NORM_IGNORECASE for exsample so I get nothing I can just enter this manually!=?But why?Kernel lib & inc are in the header above.Strange.

 

greetz

[White]

1. A loop with InternetQueryDataAvailable

	do 

	{

		dwTotalWrite	+=dwSize;

	} while (InternetQueryDataAvailable(hUrlFile,&dwSize,0,0));

2. find string for opcodes,it is better using strstr command.

	char	str[]="abcdefgh";

	char	*str1=strstr(str,"f");

	printf("%s",str1);
	output:fgh

Edited November 1, 2014 by White、、

[LCF-AT]

Hi,

 

ok listen.I have a problem with the InternetReadFile API so I still get not the full source site back into memory.Mostly the first time I get some 1000 - 2000 bytes back and the rest is missing but if I try it again right after the first time (same again InternetOpenUrlA & InternetReadFile) then I get the full lenght of site source.The question now is why there is a problem with the API?So this sucks again!So now I have to read the memory and check the end for...

</body>

</html>

...for exsample which is the end of page source in my case.This is bad again so I don't wanna add tons of extra checks just because the API xy is not working correctly or whatever you know.Hmmm!Also note that I don't know the page source size of course which also gets updated each time.What now?

 

So I think the API comparestring is ok so far for me.Seems to work without problems & good speed but thanks for the strstr exsample.

 

Thanks again and maybe you have a solution for this size problem what I have and why the API does not work as it should (right?).

 

greetz

[White]

em, I am not really understand what is wrong with them,cause I am not a good coder.    Maybe you can google it for the reason,

 

I have test the part of  "InternetQueryDataAvailable", not quiet as good as I described, sorry.

this time I create a sample and the source code is below. So you can translate it to asm.

 

Example & src.rar

[LCF-AT]

Hi,

 

thanks for your exsample file but I can't load / run it so there is any msvcr110.dll & msvcp110.dll problem for me (FlsGetValue not found in kernel32).I use XP so maybe you can compile it again.

 

But also with your exsample code the API just reads always just 296 bytes via InternetQueryDataAvailable but the site xy has some 1000 bytes.Hmmmmm!

 

Listen: The manually way is so simple....enter link xy in browser / press strg+U (open source page) / copy entire site to text file = Done. No idea why the APIs have again any trouble to do this correctly so maybe you know any other way which could work better without problems etc.

 

Thank you

 

EDIT: Ok wait....so can I also just repeat the InternetReadFile again & again till lpdwNumberOfByteRead is set to 00?So I have test it a little and this could or seems to work so after a few times the bytes getting more and then it will set to 00 but now I have the entire source into.

Edited November 1, 2014 by LCF-AT

[LCF-AT]

Hi again,

 

I have a another question about the InternetOpenUrl API.So I see this API need always some little time (2-3 seconds) each time if I call any link.So is there a way to call this API faster etc?So the problem is that I have to call very much links (500 and later much more) and for each link I have to use this InternetOpenUrl API which takes a long time.So what can I do in this case to make it faster etc?So at my test app I did grab 500 links and the app dose need 5-6 minutes till it has checked them all.Thats not so good. Hmmm.

 

Just imagine I call any internet site which has many under sites 

hompage/_001.html

hompage/_002.html

hompage/_003.html

.....

hompage/_500.html

 

and each site I have to check for strings xy (if found on site then log to file later).Maybe you have any idea whether its needed to call each time the InternetOpenUrl or not or you have any other ideas etc.

 

Thank you

[atom0s]

If you do not need to post any data to the url, you could connect to the url manually using raw sockets.

Which should remove some of the overhead created by the InternetXXX functions.

However you will still have the delay of the page loading on your connection speed regardless of what API you use. So there is going to be some delay no matter what.

[White]

 

 

can't load / run it

You need M.S visual C++ runtime library 2012 ,just install it, about 1.68 MB.

 

 

 

just reads always just 296 bytes via InternetQueryDataAvailable

Of course, you can just delete that part and just use InternetReadFile to read source and dwSize set to a large number till lpdwNumberOfByteRead is 0.

 

 

 

InternetOpenUrl.... make it faster

em,I have no idea with it, in my case, each time it takes about 400 ms.

[LCF-AT]

Hi,

 

good ok so it seems that I have to live with the delay issue of InternetOpenUrl API. Also I have no ideas about raw sockets etc (VB stuff right?).Are these working faster anyhow?So as I said I only want to read the page sources.

M.S visual C++ runtime library 2012

So I have already the msvcr110.dll & msvcp110.dll.Anyway,so can you post a link where to download this pack on ms?

 

Yes so I think only to use the InternetReadFile API in a loop till lpdwNumberOfByteRead is set 0 seems to be the better solution instead to use InternetQueryDataAvailable API.

 

No idea why InternetOpenUrl takes so long time in my case. My internet connection is 1.1 MB/s download.Maybe its to less and you have any higher download speed.

 

Ok thanks again for your help guys.

 

greetz

[White]

http://www.microsoft.com/en-us/download/details.aspx?id=30679

 

Yes, InternetReadFile is better .

[LCF-AT]

Hey White,

 

thanks for the link now your file works. I have check it now and it works almost for 100 %. Your file does download the t4y site till line 7156

<a rel="nofollow" href='https://forum.tuts4you.com/priva

but it has 7254 lines...

</html>

...anyway so then I just keep to use InternetReadFile only to get all datas.

 

Thanks again.

 

EDIT: One question.So why I always get a ERROR_INVALID_HANDLE after using InternetCloseHandle API?

EAX 00000001

ECX 001C5500

EDX 000E0001

EBX 00CC000C

ESP 0012FA84

EBP 0012FAA0

ESI 408C90CA wininet.InternetCloseHandle

EDI 0178F890

EIP 004015C1 Example.004015C1

C 0  ES 0023 32bit 0(FFFFFFFF)

P 1  CS 001B 32bit 0(FFFFFFFF)

A 0  SS 0023 32bit 0(FFFFFFFF)

Z 1  DS 0023 32bit 0(FFFFFFFF)

S 0  FS 003B 32bit 7FFDE000(FFF)

T 0  GS 0000 NULL

D 0

O 0  LastErr ERROR_INVALID_HANDLE (00000006)

EFL 00000246 (NO,NB,E,BE,NS,PE,GE,LE)

greetz

Edited November 4, 2014 by LCF-AT

[White]

em, I aware that there is something "wrong" with InternetQueryDataAvailable before posting when the first time I use that api to get available size, I also had checked it by google and found several topics poster with the same problem,but forgotten the solution.  

 

About the InternetCloseHandle api,I checked the source again ,and set a code before and after it. <dwError =GetLastError();>

Found dwError is always 0.

 

 

Edited November 5, 2014 by White、、

[LCF-AT]

Hehe again any strange API issue right. Anyway so I don't use this API anymore if the other way (read API) does also work in a loop.

 

About InternetCloseHandle API.So what is the problem now with this API?So I also get always a (ERROR_INVALID_HANDLE (00000006) back but in eax is also value 1 = succes right?

ERROR_INVALID_HANDLE
The handle that was passed to the API has been either invalidated or closed.

Ok listen,so I see the problem should be the InternetReadFile API so after using this API and then the

InternetCloseHandle = ERROR_INVALID_HANDLE for both handles of InternetOpenA & InternetOpenUrlA API.So if I don't use the InternetReadFile API and call direct the two InternetCloseHandle APIs then I get a success for both.So it seems that there should be any another API (InternetCloseFile maybe etc no idea) to use before or?

Below a quick exsample....InternetCloseHandle only works if I not call InternetReadFile....

    Invoke InternetOpen,addr szAgent,INTERNET_OPEN_TYPE_PRECONFIG,0,0,0

    mov InternetHandle,eax
    Invoke InternetOpenUrl,InternetHandle,addr szUrl,0,0,0,0

    mov FileHandle,eax

;;-------------

    Invoke InternetReadFile,FileHandle,SECTION,100000h,addr BytesRead

;;-------------
    invoke  InternetCloseHandle, FileHandle

    invoke  InternetCloseHandle, InternetHandle

...but the question is why it fails?Do you know the reason and how to fix it etc?

 

greetz

[LCF-AT]

Hi its me again so I forgot something to ask before about a new problem I have.

 

Question: How to sort lines into a alphabetical order before write the export text file?So I mean is there any simple way / API etc what can do this to prevent a manually letter / nummer / sings checking?So I have already logged all text datas into a memory block which I can write into text file now but its not in a alphabetical order.....

 

Exsample:

Hammer

Sichel

preis

Zanken

Ablesen

So this I get now into my tex file each line any word.But I wanna export it in that way..

Ablesen

Hammer

preis

Sichel

Zanken

...but how to handle this?Maybe you know also a simple solution for this (APIs / codes / whatever can help).Sorry for asking again but "correctness must be"

 

greetz

[White]

About InternetCloseHandle API

 

Have you checked the sample ? Does it always get ERROR_INVALID_HANDLE when calling InternetCloseHandle ?

I have checked the source code again,and found no problem.

 

 

 

 

MSDN remarks Returns TRUE if the handle is successfully closed, or FALSE otherwise.

The function terminates any pending operations on the handle and discards any outstanding data. If a thread is blocking a call to Wininet.dll, another thread in the application can call InternetCloseHandle on the Internet handle being used by the first thread to cancel the operation and unblock the first thread.

If there is a status callback registered for the handle being closed and the handle was created with a non-NULL context value, an INTERNET_STATUS_HANDLE_CLOSING callback will be made. This indication will be the last callback made from a handle and indicates that the handle is being destroyed.

If asynchronous requests are pending for the handle or any of its child handles, the handle cannot be closed immediately, but it will be invalidated. Any new requests attempted using the handle will return with an ERROR_INVALID_HANDLE notification. The asynchronous requests will complete with INTERNET_STATUS_REQUEST_COMPLETE. Applications must be prepared to receive any INTERNET_STATUS_REQUEST_COMPLETE indications on the handle before the final INTERNET_STATUS_HANDLE_CLOSING indication is made, which indicates that the handle is completely closed.

An application can call GetLastError to determine if requests are pending. If GetLastError returns ERROR_IO_PENDING, there were outstanding requests when the handle was closed.

 

About string sort. Here is my solution,and tests good.

 

EDIT: forgot one cmp

Sort & src upt.rar

 

 

Edited November 6, 2014 by White、、

[LCF-AT]

Hi again,

 

yes I have checked your file you have send to me before "Example.exe" and there I get also 2 times a " LastErr ERROR_INVALID_HANDLE (00000006)" after stepping over InternetCloseHandle API..below the VAs of your file.Just load it into Olly and set a BP on it and then step and check what you get.Just check this in Olly to see whether you get the same trouble.

004015B9     53                PUSH EBX

004015BA     FFD6              CALL ESI                             ; wininet.InternetCloseHandle

004015BC     FF75 F4           PUSH DWORD PTR SS:[EBP-0xC]

004015BF     FFD6              CALL ESI                             ; wininet.InternetCloseHandle

About your new sort file....so I still have again any problem with this file to get it run so Windows tells me that the file is not a valid 32 file.So it seems that I also for this new file need any system add-on which I don't have installed yet.So what do I need now again?

 

So can I use your source code also in WinASM?I don't think so or?

 

Thanks again so far