There are a number of ways to find the origin IP address of a websites server. When this is the case, you can query the origin server with a tool like curl or Postman which allows you to set HOST headers or add a static mapping to your hosts file.įinding The IP Address of the Origin Server Sometimes accessing the website via the origin IP address by inserting it in your browsers address bar won't work, as the server may be expecting a HTTP HOST header. Once you find this IP address, you can configure your scrapers to send the requests to this server instead of Cloudflares servers which have the anti-bot protection active.įor example, the origin IP address of, a Cloudflare protected site is publically accessible: Make mistakes when setting up their website on Cloudflare.īecause of this, sometimes with a bit snooping around you can find the IP address of the server that hosts the master version of the website.Here instead of having to trick Cloudflare into thinking your requests are from a real user, you instead bypass Cloudflare completely by finding the IP address of the origin server that hosts the website and send your requests to that instead.Ĭompletely bypassing Cloudflare and all its protections!Ĭloudflare is a sophisticated anti-bot protection system, but it is setup by humans who: It isn't always possible, but one of the easiest ways to bypass Cloudflare is to send the request directly to the websites origin servers IP address instead of to Cloudflare's CDN network. Option #1: Send Requests To Origin Server Option #6: Reverse Engineer Cloudflare Anti-Bot Protection.Option #5: Smart Proxy With Cloudflare Built-In Bypass.Option #4: Scrape With Fortified Headless Browsers.Option #1: Send Requests To Origin Server.So in this guide, we're going to go through each of those options so you can choose the one that works best for you. They range from the easy like using off-the-shelf tools, to the extremely complex like completely reverse engineering how Cloudflare detects and blocks scrapers. There are a number of approaches you can take to bypassing Cloudflare, all with their own pros and cons. Luckily for us, bypassing Cloudflares anti-bot protection is possible. With an estimated 40% of websites using Cloudflares Content Delievery Network (CDN), bypassing Cloudflare's anti-bot protection system has become a big requirement for developers looking to scrape some of the most popular websites on the internet.

Octoparse captcha