title

How many requests per second is reasonable when scraping web data?

name

Sandra Pique

01-14-2022

It really depends on the website , the infrastructure, how sensitive the website is to scrapers, the normal traffic, and many other things. Generally, the more traffic the website has the more frequently you can access it.

 A good practice is to keep number of requests at approximate 400 per hour.If that is not fast enough for you, I’d suggest cheking some proxy providers for that matter, as it it possible to spread out your requests to different ip.Use proxy services while scraping, since your IP address is constantly changing because of the rotating proxies, as well as it imitates human activity online and prevents you from being detected by the website.

If the IP needs to be an e-commerce platform or social media, consider selecting roxlabs dedicated computer room IP. Fast IP, easy to set, unlimited traffic.

More on: Roxlabs proxy 

Recent posts

title
username

Sandra Pique

How to get their Instagram bot?
01-10-2022
title
username

Sandra Pique

what do proxies do for supreme?
01-18-2022