Scraping routers, and when routing beats picking

Seven services that route a request rather than serve it themselves. Each one sits in front of several backends and picks per request, so the question is never whether routing works but what it costs you in visibility when it does.

• ScrapeOps Proxy Aggregator

  Free tier, then paid

  One endpoint in front of a large set of third party proxy providers, choosing among them per request and falling through to another on failure.

  Suits

  The closest thing to a pure aggregator on this list. You keep one integration while the pool behind it changes, and the dashboard reports success rate and cost per provider, which is the number that usually decides whether a provider is worth keeping.

  Does not suit

  It aggregates proxies, not rendering or parsing, so anything needing a real browser still needs a separate answer. You are also trusting its per provider measurements, and they are the same measurements that decide which provider it sends you to.

• Zyte API

  Paid

  A single endpoint that decides per request how much machinery a page needs, from a plain fetch up to a full browser session with ban handling.

  Suits

  The decision is the product. It starts cheap and escalates only when a page actually requires it, which on a mixed crawl costs less than sending everything through a browser because a few pages need one.

  Does not suit

  The escalation logic is not yours to inspect or override in detail, so a page that gets misjudged is hard to argue with. Pricing follows what it decided to do rather than what you asked for, which makes a bill harder to predict from a URL list.

• spider.cloud

  Free tier, then paid

  A crawling API that routes a request across its own fetch strategies and third party providers, exposing one interface over several backends.

  Suits

  Routing extends to providers outside its own infrastructure, so a target that one backend cannot reach can be served by another without a second integration. Output formats aimed at downstream text processing remove a conversion step for pipelines feeding a model.

  Does not suit

  Smaller and younger than the enterprise vendors here, so the published track record on hostile targets is thinner and the support surface is smaller. Provider routing also means your effective cost and success rate depend on backends it chose, not ones you picked.

• Bright Data Web Unlocker

  Paid

  A managed endpoint that handles proxy selection, browser rendering and challenge solving behind one request, over a very large address pool.

  Suits

  The pool is the largest here by a wide margin, which shows on targets that block by network reputation. Compliance and provenance documentation is more developed than anywhere else on this list, which matters if procurement is involved.

  Does not suit

  The largest contract and the most complexity. Pricing has several axes and the cheapest correct configuration is genuinely hard to find, so teams routinely overpay for months before tuning it. Overkill for targets that serve plain HTML.

• Oxylabs Web Unblocker

  Paid

  An unblocking layer that picks a proxy type, headers and rendering path per request, sitting in front of the same residential and datacenter pools sold separately.

  Suits

  Session and geo control is more explicit than most routers expose, so a target that behaves differently by country is testable rather than guesswork. Documentation names what it tried, which shortens debugging.

  Does not suit

  Priced for teams with a procurement process, and the entry tiers are high enough that a small project is better served elsewhere. Routing is across its own products rather than across vendors, so it does not reduce vendor concentration.

• Apify

  Free tier, then paid

  A platform where scrapers published by many authors run on shared infrastructure, with proxy selection handled underneath.

  Suits

  Routing at the scraper level rather than the request level. For a popular target somebody has usually already written and maintained the extractor, which skips the part of the work that actually decays.

  Does not suit

  Quality varies by author, and a marketplace actor can be abandoned with no notice while still appearing to work until the target changes. You are depending on a third party's maintenance, which is the thing you were trying to buy.

• Scrapoxy

  Open source · MIT · TypeScript

  Self hosted software that aggregates proxy providers and cloud instances behind one local endpoint you run yourself.

  Suits

  The only entry here where the routing logic is yours to read and change, and credentials stay on your infrastructure. Cost is whatever the underlying providers charge with no margin on top.

  Does not suit

  You operate it, which means the availability of your scraping is now your own uptime problem. It routes proxies only, so rendering, parsing and challenge handling remain entirely unsolved.

A router is a service you hand a URL to, which then decides which backend actually fetches it. That decision is the whole product. Everything else on the pricing page follows from it.

The case for routing is straightforward and mostly about maintenance rather than capability. No single backend wins on every target, and holding three vendor accounts means three SDKs, three billing relationships and three sets of failure modes to learn. A router collapses that into one integration. What you give up is knowing which backend served any particular request.

The question to ask before you depend on one

What does a failed response tell you about the path it took?

This is the difference between a bad afternoon and a bad week. When a request fails through a router, you need to know which backend handled it and why that one was chosen. If the response says neither, your debugging loop now runs through somebody else's support queue, and you are reduced to guessing from timing and error codes. Some routers report the provider and the strategy on every response. Some report nothing. That distinction is worth more than a few percent of success rate, and it is rarely on the comparison table.

The second question is what happens when the router is down rather than a backend. Several backends behind one endpoint is genuinely more resilient than one backend, until the router is the failing component, at which point every target is unreachable simultaneously rather than one of them being difficult. If the pipeline matters, keep a direct path to at least one backend and test it occasionally.

Where routing does not help

It does not defeat fingerprinting. Rotating addresses helps against rate limits and crude reputation scoring, because those decisions are made about the address. A defence that has already concluded your client is automated reached that conclusion from the client: the TLS handshake, the header order, the absence of the things a real browser does. Routing to a different address changes none of that and the block follows you.

It does not help on a stable target. If one site serves plain HTML to anything that asks, a router adds a hop, a bill and a dependency to a problem that a direct request already solved.

It does not fix selectors. Every service here returns a page or a fetch result. When the markup changes, the extraction breaks, and nothing in the routing layer notices, because from its side the request succeeded. See robots.txt for what a crawler is allowed to ask for, and 429 for what a target says when you ask too often.

Checking what a target actually returns first

Most routing decisions are made before anyone has established what the target does to an ordinary request. A page that serves complete HTML to a plain fetch needs no rendering, and paying per request for a browser that renders it again is the most common avoidable cost in this category.

Comparing the raw and the rendered response for one representative URL answers that in about a minute, and the answer usually holds for a whole section of a site. The rendering comparison covers how to read the two responses. How a crawler resolves a page covers the steps that run before rendering is on the table.