Help

Latest News From Help

Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.

How to use IP to speed up your web data fetching?

When multiple requests are made to a site using the same IP address, the site can mark your IP as a crawler's IP, i.e. an abnormal user, and block your IP or send misleading information so that you can't crawl properly. So regularly changing or switching your IP address can help prevent target sites from identifying your IP as a crawler. Cyclic IP switching can improve the success rate of capture. It can be operated by Proxy Manager software with the IP extracted by RoxLabs:You can change the number of requests that can be sent before an IP address and the IP switchover interval.

Read More +
FAQ
12-10-2021

How to avoid being blocked IP by scraping website data?

When a target site detects abnormal user access to the site (frequency and duration of operation detected and proxy IP detected), it usually:1. Mask the IP address2. Display misleading information to the IP address3. Slow down the response timeTarget sites log the IP addresses of visitors and analyze the activity of those IP addresses. Assuming you are using a traditional data center proxy or a low-anonymity proxy, the target site can do the following:The ability to identify activity (request rate) from a single IP in a given period of time is much higher than the human abilityIdentify IP addresses from the list of proxy servers that these target sites can accessIdentify these IP addresses as having the same subnet mask rangeHow to avoid detection?To avoid detection due to the number of requests per IP, you can reduce the number of requests per second. However, this can also slow down your data fetching speed. In order to avoid being detected by the target website because your IP came from a proxy server, you must cycle your requests through different residential IP addresses. You should be able to loop through enough ips to ensure that the target site cannot detect your activity. With residential IP, there is no subnet blocking range, residential IP proxy with a legitimate IP address, will not blacklist you from the site. In the case of a data center proxy, the site owner can detect that it belongs to the data center and not the ISP.

Read More +
FAQ
12-10-2021

What should I do if I don't receive an email confirmation from RoxLabs after registering?

Check your spam mailbox. If it's not in your spam mailbox, try sending it again and wait a few minutes to see if you got it.If not, please contact us at support@roxlabs.io and we will get back to you soon.

Read More +
FAQ
12-10-2021

What crawler proxy to choose?

What proxy servers should be selected based on the source of the crawler? In the previous section, there are two types of proxy servers addresses: residential proxy servers address and equserversment room proxy servers address. They have their own characteristics. Let's have a look. 1. Residential agent servers is usually unconnected, with scattered regional distribution, high success rate, but poor stability 2. The agent servers of the machine room is generally connected, the regional distribution is relatively concentrated, the stability is high, but the success rate is relatively low. From the price, the machine room agent is cheaper. Therefore, in order to save cost, crawler work should choose the computer room servers agent with lower price, rather than the residential agent servers, so is it really true? In fact, the best way to crawler is to choose servers. What's wrong with this is that there are many disadvantages to using machine room proxy servers. When using the machine room proxy servers crawler, the target server can identify whether your servers is a proxy servers or not, but because of the high privacy of proxy servers, the server cannot know your real servers address. In this case, the server will have two feedback: the first is a direct denial of the access request; The second is to receive the request but give you the processed data, that is, only the information the server wants you to see. Obviously, in either case, reptiles don't want to see it, especially in the second case, which is terrible. This problem does not occur with residential proxy servers. Because the target server does not detect the residential proxy servers, the crawler is also used smoothly. Therefore, the crawler's work is best to choose the residential proxy servers.If you need multiple different proxy IP, we recommend using RoxLabs proxy：https://www.roxlabs.io/, including global Residential proxies, with complimentary 500MB experience package for a limited time.

Read More +
FAQ
11-10-2021

Which proxy is better for use with sneaker robots?

Once sneaker sites find out you're using bots, the first thing they do is block IP addresses. The proxy solves this problem by providing multiple IP addresses. However, proxies must be used correctly. If you use the wrong proxy service, you may end up not getting sneakers and your investment will be wasted.In general, sneaker robots have two types of proxies, data center proxies and residential proxies.Secondary companies provide data center proxies. The IP they provide is a combination of numbers, not real. Instead, the residential proxy provides the raw IP address provided by the Internet service provider (ISP). These IP addresses are associated with physical locations to make them appear legitimate.The risk of being detected using a data center proxy is relatively small, but if you want to keep your chances to zero, residential IP proxies are best suited. Data center proxies are easy to detect because most of the IP they provide uses a single subnet. If the retailer identifies an IP, it blocks the entire subnet. In the case of residential proxies, they are almost never blacklisted because they are associated with isPs. Therefore, it is more recommended to use residential proxies for sneaker robots. Roxlabs offers global regional residential proxy services, 500MB test for new users, welcome to Roxlabs.

Read More +
FAQ
11-10-2021

How to prevent IP from being blacklisted?

Blacklisting of IP addresses can happen frequently when fetching web data. Here are some of the best ways to prevent being blacklisted while crawling:1. Using IP rotation service, RoxLabs provides a set of IP to crawl the network. This will avoid sending so many requests using the same IP address and keep your IP secure.2. Set up popular user proxies for web scraping tools such as Google, Microsoft, etc. Doing so can trick the site into believing that you are visiting their site as a real user. Often, fetching robots forget to display user proxies  and are easy to catch.3. Try to crawl like a normal user, such as crawling sites 24 hours a day, because normal users will never do this.Add a referral source url like Google, YouTube or Facebook to your request so that the site owner knows where you're coming from. This will make identifying your request more straightforward, and the site will feel like you are a real user.Some clever webmasters add honeypot traps to detect crawlers and bots. Your scraping tools and proxies should avoid such traps by browsing the site as a real user and avoiding clicking on hidden links.

Read More +
FAQ
11-10-2021