Resources

One list per category. Every entry names a limitation, because a row that only says good things is an advertisement and reads as one.

• Proxy providers, and where each one stops working

  Fifteen commercial proxy networks, each with the case it fits and the case it does not. Pricing shape matters more than pool size for most jobs, so both are named.

  15 entries

• Crawlers, from site audit tools to crawl frameworks

  Seventeen crawlers, split between tools that audit a site you own and frameworks that crawl the web at large. The two groups fail in different ways, so both are marked.

  17 entries

• Scraping APIs and tools, with their real limits

  Fifteen services that fetch a page and hand back data. The split that matters is whether the vendor maintains the selectors or you do, so each row says which.

  15 entries

• Browser automation APIs and hosted browsers

  Eighteen ways to drive a real browser, from the protocol underneath to hosted fleets you rent by the minute. Memory is the recurring constraint, so each row names its cost.

  18 entries

• Browser agents, and what they decide for you

  Six tools that read the page and choose the next click themselves, instead of running the clicks you wrote. The model in the loop is the feature and the bill, so each row names both.

  6 entries

• Search APIs, and what each index actually covers

  Fourteen ways to query the web programmatically. They divide into three groups that answer different questions, and mixing them up is the common and expensive mistake.

  14 entries

• Scraping routers, and when routing beats picking

  Seven services that route a request rather than serve it themselves. Each one sits in front of several backends and picks per request, so the question is never whether routing works but what it costs you in visibility when it does.

  7 entries

• Open source crawling and extraction projects

  Every license here was read from the project's own repository rather than from a directory. Two of these are copyleft in a way that reaches your code, and those rows say so.

  22 entries

What these are

Static reference pages. Nothing here runs a crawl, and no entry was collected by one. Each row was written by hand after reading the vendor's own documentation, and every link was fetched to confirm it resolves before the page was published.

The limitation field is the reason these exist. Anyone can list the vendors in a category. Knowing which one stops working when your target starts checking TLS fingerprints, or which auditor caps a free crawl at 500 URLs, is the part that saves an afternoon.

No entry is ranked above another and nothing here is a recommendation. Where a tool suits a job, the row says which job. Where it does not, the row says that too.

Check the target before you pick a tool

Most tool decisions in this area are made before anyone has established what the target actually does. Fetch the URL twice, once plain and once with a browser, and compare the status code and the byte count. The rendering comparison covers how to read the difference, and how a crawler resolves a page sets out the order the steps run in.

Two results change which list you need. If the page comes back complete without JavaScript, you do not need a browser. If it comes back with a 403 rather than content, you have a block to understand rather than a parser to write, and robots.txt is the first place to look.