Jump to content
Tuts 4 You

Help needed in web scrapper


kate

Recommended Posts

Hi everyone,
       I need your help and suggestions in one of my testing projects. I'm trying to create one web scrapper to purchase things in flash sale on low rate from various ecommerce sites. I have successfully crated all things and all steps but getting very slow response during the load. Can anyone guide me how I can get the faster response from server even if its responding slow. For creating scrapper I have first used the PHP. And currently I'm trying with C#.net. Any suggestion to improve performance. Please send me your valuable comments.

 

Thanks

Link to comment

you can use :

Visual.Web.Ripper 

http://visualwebripper.com/

 

Content Grabber

https://contentgrabber.com/

is not free ofc :)

  • Like 1
Link to comment

Implement your own browser and disable a lot of the heavy loading things like images and unneeded scripts. There are multiple browsers that are open source, such as Chromium (CEF) which you can look into to implement things like this. It'll help speed up page loads as you wont be loading images every refresh / navigation request. Ensure that you have caching on as well for things that you do allow to load so they do not need to be reloaded each request. 

Link to comment
3 hours ago, atom0s said:

Implement your own browser and disable a lot of the heavy loading things like images and unneeded scripts. There are multiple browsers that are open source, such as Chromium (CEF) which you can look into to implement things like this. It'll help speed up page loads as you wont be loading images every refresh / navigation request. Ensure that you have caching on as well for things that you do allow to load so they do not need to be reloaded each request. 

Can I Use HTTP Web Request instead ?

Link to comment

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...