Hi everyone,
       I need your help and suggestions in one of my testing projects. I'm trying to create one web scrapper to purchase things in flash sale on low rate from various ecommerce sites. I have successfully crated all things and all steps but getting very slow response during the load. Can anyone guide me how I can get the faster response from server even if its responding slow. For creating scrapper I have first used the PHP. And currently I'm trying with C#.net. Any suggestion to improve performance. Please send me your valuable comments.

 

Thanks

you can use :

Visual.Web.Ripper 

http://visualwebripper.com/

 

Content Grabber

https://contentgrabber.com/

is not free ofc

Implement your own browser and disable a lot of the heavy loading things like images and unneeded scripts. There are multiple browsers that are open source, such as Chromium (CEF) which you can look into to implement things like this. It'll help speed up page loads as you wont be loading images every refresh / navigation request. Ensure that you have caching on as well for things that you do allow to load so they do not need to be reloaded each request. 

3 hours ago, atom0s said:

Implement your own browser and disable a lot of the heavy loading things like images and unneeded scripts. There are multiple browsers that are open source, such as Chromium (CEF) which you can look into to implement things like this. It'll help speed up page loads as you wont be loading images every refresh / navigation request. Ensure that you have caching on as well for things that you do allow to load so they do not need to be reloaded each request. 

Can I Use HTTP Web Request instead ?

WebRequest should work fine as well.