Scraping APIs and tools, with their real limits

Fifteen services that fetch a page and hand back data. The split that matters is whether the vendor maintains the selectors or you do, so each row says which.

• ScraperAPI

  Free tier, then paid

  A request proxy that handles rotation, retries and optional browser rendering, returning the page HTML from a single endpoint.

  Suits

  The integration is one URL swap, so an existing crawler can be pointed at it without restructuring. Failed requests are generally not billed, which changes the arithmetic on a difficult target.

  Does not suit

  It returns HTML, not data. Parsing, selector maintenance and the schema are all still yours, and the structured endpoints that do parse cover a fixed set of popular sites rather than yours.

• ScrapingBee

  Free tier, then paid

  A rendering and proxy API that returns HTML, screenshots or extracted fields via a CSS selector rule set passed with the request.

  Suits

  Small and well documented enough to integrate in an afternoon, with JavaScript rendering and a declarative extraction option that removes a parsing step for simple pages.

  Does not suit

  Rendered requests consume several times the credits of plain ones, so cost tracks how carelessly you enable rendering. The extraction rules are shallow, and anything needing conditional logic or pagination state goes back into your own code.

• Scrapfly

  Free tier, then paid

  A scraping API with anti-bot bypass, browser rendering, session handling and a hosted extraction layer.

  Suits

  The debugging surface is unusually good: each request logs what was tried, what the target returned and which bypass path ran, which turns a silent failure into something you can read.

  Does not suit

  A per-request pricing model with feature multipliers makes cost hard to forecast before you have run real traffic. Budget a test batch specifically to learn what your target costs.

• Zyte API

  Paid

  A single API covering proxy selection, browser rendering and automatic extraction, from the team behind Scrapy.

  Suits

  The automatic extraction models return typed fields for common page shapes such as products and articles without any selectors, which survives a redesign that would break a hand-written parser.

  Does not suit

  Automatic extraction works on the page types it was trained for. Anything outside them falls back to raw HTML and your own parsing, and the ban-handling that justifies the price is wasted on targets that never blocked you.

• Apify

  Free tier, then paid

  A platform for running scrapers, with a marketplace of prebuilt ones and a runtime for your own code.

  Suits

  Someone has usually already written and published a scraper for a popular site, so the first working run can be minutes away. Your own code gets scheduling, storage, proxies and logs without you building any of it.

  Does not suit

  Marketplace scrapers are third-party code with third-party maintenance, and a popular one going stale is not your bug to fix but is your outage. Pricing combines platform credits with per-actor charges, which is two meters to watch.

• Firecrawl

  Free tier, then paid

  A crawl and scrape API that returns page content as clean Markdown, aimed at feeding language models and retrieval pipelines.

  Suits

  Markdown output that strips navigation and boilerplate saves a preprocessing step for anything that ends up in a prompt or an index, and the same codebase is available to self-host.

  Does not suit

  Markdown is lossy on purpose. Tables, nested structure and any attribute you needed, such as a price in a data attribute, may not survive the conversion. Do not use it when the data lives in the markup rather than the prose.

• spider.cloud

  Free tier, then paid

  A crawl, scrape and search API that returns page content as Markdown or JSON, streamed as pages arrive.

  Suits

  It fetches over plain HTTP first and starts a browser only for pages that need one, so a mixed crawl does not pay rendering cost on the pages that never required it.

  Does not suit

  There is no structured extraction. It returns content, and the schema and the selectors stay yours. Billing meters bandwidth and CPU time rather than requests, so no plan tells you what a URL costs until you have run one.

• Diffbot

  Paid

  An extraction service that classifies a page and returns typed fields for it, plus a knowledge graph built from its own crawl.

  Suits

  Page classification and extraction happen without any per-site configuration, so a heterogeneous list of thousands of unrelated URLs comes back in one schema.

  Does not suit

  You cannot correct a field you disagree with. Extraction is a model decision, not a selector you can edit, so when it misreads an unusual layout the fix is a support conversation rather than a code change.

• Bright Data Web Scraper API

  Paid

  Prebuilt per-site scrapers and a general unblocking endpoint, sold on top of the same network as the company's proxies.

  Suits

  The maintained scrapers for large commerce and social targets absorb the breakage that would otherwise land on you every time one of those sites redeploys.

  Does not suit

  Coverage is a catalog. A site outside it drops you back to the generic endpoint and your own parsing, and the compliance review that applies to the proxy products applies here too, so access is not immediate.

• ZenRows

  Free tier, then paid

  A scraping API bundling proxy rotation, headless rendering and anti-bot handling behind one request.

  Suits

  Positioned tightly at the blocked-request problem, so the defaults are already tuned for targets that return a challenge page rather than content.

  Does not suit

  That focus is the limit too. On targets that were never blocking you, you are paying an anti-bot premium for a fetch that curl would have completed, and extraction remains your job.

• Octoparse

  Free tier, then paid

  A desktop and cloud scraper where you build extraction rules by clicking elements on a rendered page.

  Suits

  Non-programmers can build a working scraper for a paginated list in an afternoon, including login steps and scheduled cloud runs.

  Does not suit

  Point-and-click rules are hard to review, hard to diff and hard to repair in bulk. When a site redesign breaks forty tasks, you are reopening forty tasks by hand, because there is no shared selector to fix once.

• ParseHub

  Free tier, then paid

  A visual scraper that handles interactive pages, with a free tier and hosted runs.

  Suits

  Its relative selection model copes with nested and irregular layouts better than most click-to-select tools, and the free tier is generous enough for a real small project.

  Does not suit

  Run speed on the free and lower tiers is slow enough that anything time sensitive is off the table, and the project has moved slowly, so newer anti-bot defenses are not what it was built for.

• Browse AI

  Free tier, then paid

  A recorder that turns a browsing session into a repeatable extraction robot, with monitoring and change alerts.

  Suits

  Recording a flow through a login and a search form is genuinely faster than writing it, and the monitoring product turns a scraper into a change alert without extra plumbing.

  Does not suit

  A recorded flow is brittle by construction: it encodes the path you took, so an A/B test or a moved button breaks it. It also suits page-level extraction rather than a crawl of a hundred thousand URLs.

• Web Scraper

  Free tier, then paid

  A browser extension that builds sitemaps of selectors for point-and-click extraction, with an optional cloud runner.

  Suits

  The extension is free and runs entirely in your own browser, which makes it the cheapest way to pull a few hundred rows out of a site once, with no account and no data leaving your machine.

  Does not suit

  Local runs are bounded by one browser tab and stop when you close it. Scheduling, parallelism and proxy support only arrive with the paid cloud service, which is a different product with different constraints.

• Import.io

  Paid

  A managed web data platform focused on retail and pricing intelligence, sold with a service layer rather than as a self-serve API.

  Suits

  The vendor takes on extraction and maintenance as part of the contract, which suits teams that want the data feed and explicitly do not want to own scrapers.

  Does not suit

  Managed means gated. Onboarding runs through sales, turnaround on a new source is their schedule rather than yours, and a self-serve experiment is not really on offer.

What you are actually buying

Every service here does some subset of four jobs: get the bytes, run a browser, find the fields, keep it working when the site changes. The price differences are mostly about how many of those four the vendor keeps.

A fetch API does the first two. You still write the selectors and you still own the 3am break when a class name changes. A structured extraction service does all four for the page types it knows, and hands the whole thing back the moment your target sits outside them. That boundary is worth finding before you sign anything, and finding it costs one test batch of your real URLs.

The failure nobody instruments

Scrapers do not usually crash. They return a row with three good fields and one null, over and over, and the pipeline downstream accepts it because null is a valid value. Weeks later somebody notices the price column has been empty since a deploy in the spring.

The fix is cheap and almost nobody does it: assert on fill rate, not on exit code. If a field is populated on 98 percent of rows in normal operation, a run where it drops to 4 percent should fail loudly rather than write. Same for a page that suddenly returns 200 with 900 bytes where it used to return 60,000, which is the signature of a challenge page being stored as content. A soft 404 has the same shape.

Check the target before you pay for the hard case

Most of the premium in this category is anti-bot handling, and a good share of the sites people buy it for were never blocking them. Fetch the URL plainly, then fetch it with a browser, and compare the status code and the body length. If both come back the same, you need a parser and a polite request rate, not a bypass product.

The rendering comparison covers how to read the two responses and what the difference does not prove. How a crawler resolves a page sets out where rendering falls in the order. If the plain fetch comes back 403, that is a real block. If it comes back full of content, the expensive tier is not your problem to solve.

One thing to keep separate from cost: a crawler that identifies itself honestly is a different thing from one built to look like a person. GPTBot and Googlebot publish their user agents precisely so a site can decide. Cloaking is what happens on the other side of that line.

Related

Proxy providers covers the network layer under these services. Browser APIs covers rendering if you run it yourself. Open source projects lists the extraction libraries you would use to parse whatever comes back. Scraping routers covers services that choose among several of these per request instead of leaving the choice to you. Browser agents covers the other end of the range, where a model works out the clicks at run time and nobody writes a selector.