How to solve the error of web page data scraping using an proxy?


Sandra Pique


Today, I sorted out several error codes in using proxies

1、407 Proxy Authentication Required.

The proxy authentication information is wrong. The proxy needs user authentication and needs to bring the correct user authentication header.

2、429 Too Many Requests.

There are two possibilities to return this status code:

The request is too fast. You need to reduce the request rate

The target website has an anti crawling mechanism, which limits the requests of crawlers.

3. 403 the server refused the request.

It may be caused by the protection measures of the target website. It is recommended to upgrade the crawler strategy or replace the high-quality HTTP proxy. The operator is authorized to build its own computer room with low latency and high availability.

4、504 Proxy Gateway TimeoutLink.

There are two situations when returning to 504:

The proxy is switching IP. Take a break and try again;

The target site is unreachable.

Recent posts