Jump to content
Tuts 4 You
Sign in to follow this  
kate

Help needed in web scrapper

Recommended Posts

kate

Hi everyone,
       I need your help and suggestions in one of my testing projects. I'm trying to create one web scrapper to purchase things in flash sale on low rate from various ecommerce sites. I have successfully crated all things and all steps but getting very slow response during the load. Can anyone guide me how I can get the faster response from server even if its responding slow. For creating scrapper I have first used the PHP. And currently I'm trying with C#.net. Any suggestion to improve performance. Please send me your valuable comments.

 

Thanks

Share this post


Link to post
whoknows

you can use :

Visual.Web.Ripper 

http://visualwebripper.com/

 

Content Grabber

https://contentgrabber.com/

is not free ofc :)

  • Like 1

Share this post


Link to post
atom0s

Implement your own browser and disable a lot of the heavy loading things like images and unneeded scripts. There are multiple browsers that are open source, such as Chromium (CEF) which you can look into to implement things like this. It'll help speed up page loads as you wont be loading images every refresh / navigation request. Ensure that you have caching on as well for things that you do allow to load so they do not need to be reloaded each request. 

Share this post


Link to post
Jessica
3 hours ago, atom0s said:

Implement your own browser and disable a lot of the heavy loading things like images and unneeded scripts. There are multiple browsers that are open source, such as Chromium (CEF) which you can look into to implement things like this. It'll help speed up page loads as you wont be loading images every refresh / navigation request. Ensure that you have caching on as well for things that you do allow to load so they do not need to be reloaded each request. 

Can I Use HTTP Web Request instead ?

Share this post


Link to post
atom0s

WebRequest should work fine as well. 

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  
×
×
  • Create New...