Crawlers, from site audit tools to crawl frameworks

Seventeen crawlers, split between tools that audit a site you own and frameworks that crawl the web at large. The two groups fail in different ways, so both are marked.

• Screaming Frog SEO Spider

  Free tier, then paid

  A desktop crawler for auditing a site's titles, redirects, canonicals, directives and internal link graph.

  Suits

  It runs on your machine against your own site, so there is no queue and no per-URL metering, and the crawl data stays local. The link graph and redirect chain views answer most technical audit questions directly.

  Does not suit

  The free version stops at 500 URLs and cannot render JavaScript, save a crawl, schedule a run or connect to Search Console. Memory is the other wall: a million-URL crawl in RAM mode needs a workstation, and the database mode that avoids it is slower.

• Sitebulb

  Paid

  A site auditing crawler, available as a desktop application and as a hosted cloud service, that scores findings and explains each one.

  Suits

  The output is written as prioritized advice with the reasoning attached, which makes the report usable by someone who did not run the crawl.

  Does not suit

  That opinionated scoring is a choice you are stuck with. If you disagree with how it weighs an issue, you are exporting the raw data and rebuilding the view yourself, which is more work than starting from a tool that only reports facts.

• Lumar

  Paid

  A hosted crawling and site intelligence platform, formerly DeepCrawl, aimed at continuous monitoring of large sites.

  Suits

  Scheduled cloud crawls that compare against previous runs, so a regression introduced by a deploy shows up as a diff rather than as a number you have to remember.

  Does not suit

  Priced and scoped for enterprise sites under continuous monitoring, with onboarding to match. For a one-off audit of a few thousand URLs it is more platform than the question needs.

• Oncrawl

  Paid

  A hosted crawler that joins crawl data with server log files and Search Console data in one analysis.

  Suits

  Log file analysis is the part most tools skip, and it answers questions a crawl alone cannot: which URLs search engines actually requested, how often, and what they received.

  Does not suit

  The log side only works if you can get the logs. On a stack where a CDN terminates requests and log export is a separate paid product, the feature that justifies the tool is the one you cannot feed.

• JetOctopus

  Paid

  A cloud crawler with log analysis and Search Console integration, positioned for large sites.

  Suits

  Crawl speed on large sites is its stated focus, and the log integration is included rather than sold as a separate tier.

  Does not suit

  A smaller ecosystem than the long-established platforms means fewer third-party integrations and less public writing when you hit an edge case. You will be asking their support rather than searching for an answer.

• Botify

  Paid

  An enterprise platform combining crawl, log analysis, Search Console data and its own page delivery product.

  Suits

  Built for sites with millions of URLs where the question is which templates deserve crawl budget, and the data model is built around that question rather than around a URL list.

  Does not suit

  This is a contract and an implementation, not a download. Nothing about it suits a site of a few thousand pages, and the delivery product means part of what you are evaluating is infrastructure you would be adding to your stack.

• spider.cloud

  Free tier, then paid

  A hosted crawl API that walks a whole site from one request, with concurrency, depth and sitemap controls, and returns each page as Markdown or JSON.

  Suits

  Pages stream back as they are fetched and can be written straight to S3, GCS or Supabase, so a long crawl is processed while it is still running instead of after it ends. Depth, concurrency and sitemap-only are request parameters, so the shape of a run changes in the call rather than in a dashboard.

  Does not suit

  It returns content, not an audit. The crawl-over-crawl diff, the log file join and the Search Console data that the platforms in this group are bought for are yours to build on top. Billing is in credits metered against bandwidth and compute rather than per URL, so nothing tells you what a crawl of your site costs until you have run one.

• Scrapy

  Open source · BSD-3-Clause · Python

  A Python crawling framework with its own scheduler, item pipelines, middleware layer and throttling.

  Suits

  The concurrency model, retry handling, duplicate filtering and pipeline structure are all solved and battle-tested, so a crawl that outgrows a for-loop has somewhere to go that is not a rewrite.

  Does not suit

  No browser. Anything that needs JavaScript requires bolting on Playwright or a rendering service, and the resulting hybrid loses much of the throughput that made Scrapy attractive. The Twisted-based async model also predates asyncio and reads strangely to anyone who learned Python this decade.

• Crawlee

  Open source · Apache-2.0 · TypeScript

  A crawling library for Node and Python that handles queueing, retries, proxy rotation and session management, with plain HTTP and browser crawlers behind one interface.

  Suits

  Switching a crawler from a plain fetch to a real browser is close to a one-line change, which makes it cheap to find out whether a target actually needs rendering.

  Does not suit

  The abstraction assumes its own storage layout and request queue. Fitting it into an existing pipeline that already owns queueing and state means working against the grain, and the browser path inherits every memory cost of the browser it drives.

• spider

  Open source · MIT · Rust

  A Rust crawling framework built on Tokio, with an HTTP fetcher and an optional headless Chrome path.

  Suits

  Pages stream out of the crawl as they are fetched rather than arriving in a batch at the end, so a long crawl can be processed while it is still running.

  Does not suit

  Rust is the price of entry, so a team without it pays a compile step and a borrow checker before the first page arrives. Documentation is thinner than the Python frameworks manage, and the gap shows the moment a crawl needs something the examples do not cover.

• Katana

  Open source · MIT · Go

  A fast Go crawler built for security reconnaissance, with a headless mode and JavaScript endpoint parsing.

  Suits

  Very fast URL discovery from the command line, including endpoints pulled out of JavaScript files that a link-following crawler never sees.

  Does not suit

  It is built to enumerate URLs, not to extract content, so there is no parsing or storage layer worth speaking of. Its defaults are also tuned for reconnaissance rather than politeness, which is the wrong posture for a site you do not own.

• Apache Nutch

  Open source · Apache-2.0 · Java

  A mature Java crawler designed for large-scale, distributed web crawling, with Hadoop integration and pluggable parsers.

  Suits

  It has crawled the open web at serious scale for two decades, and the plugin architecture means indexing backends and parsers can be swapped without touching the crawl loop.

  Does not suit

  The operational cost is a Hadoop-shaped cluster and a batch cycle of inject, generate, fetch, parse and update. That is the wrong shape for anything continuous, and the wrong amount of machinery for anything under a few million URLs.

• Heritrix

  Open source · Apache-2.0 · Java

  The Internet Archive's archival crawler, which writes WARC files preserving the exact request and response bytes.

  Suits

  It is built for fidelity rather than throughput, and WARC output means the archive can be replayed later exactly as it was served, headers included.

  Does not suit

  Archival fidelity is the whole design, so it is a poor fit for extraction work: you get faithful bytes, not fields. Configuration is XML-heavy and the learning curve is steep enough that most teams reach for it only when WARC output is a requirement.

• StormCrawler

  Open source · Apache-2.0 · Java

  A collection of resources for building low-latency, distributed crawlers on Apache Storm.

  Suits

  Streaming rather than batch, so URLs are fetched as they are discovered instead of waiting for a cycle boundary. That suits news and any corpus where freshness is the point.

  Does not suit

  You are running an Apache Storm topology, with everything that implies for deployment, monitoring and on-call. It is a toolkit for building a crawler rather than a crawler, so expect to write Java before you fetch anything.

• HTTrack

  Open source · GPL-3.0 · C

  A website copier that downloads a site to local disk and rewrites links so the copy browses offline.

  Suits

  One command produces a browsable local mirror, which is still the fastest way to take a snapshot of a small static site before a migration.

  Does not suit

  It belongs to the era it was written in. Anything built on client-side rendering copies as an empty shell, and the project has seen little movement in years, so modern TLS and HTTP behavior can surprise it.

• GNU Wget

  Open source · GPL-3.0 · C

  A command line downloader with recursive retrieval, present on almost every Unix machine already.

  Suits

  Zero install and completely predictable. For pulling a directory of files, mirroring a documentation tree or checking that a URL is reachable from a server, nothing simpler exists.

  Does not suit

  It is a downloader, not a crawler: no JavaScript, no queue you can inspect, no politeness logic beyond a fixed wait, and recursion controls that are easy to point at far more of a site than you meant. Check what you asked for before running it against a host you do not own.

• Common Crawl

  Free

  A nonprofit that publishes free archives of web crawl data, with page text, metadata and a URL index.

  Suits

  Petabytes of already-crawled pages, free to query, which turns many research questions into an index lookup instead of a crawl you would have to run and defend.

  Does not suit

  It is a sample, not the web, and each archive is a snapshot with its own date. Nothing in it is current, coverage of any specific site is partial and unpredictable, and the pages were fetched without JavaScript. If your question is about one site today, this is the wrong dataset.

Two different tools wearing one word

Half this list audits a site you control. The other half crawls sites you do not. They share a verb and almost nothing else.

An audit crawler assumes access. It starts from your sitemap, ignores rate limits because your server is yours, and its output is a list of problems: broken links, redirect chains, duplicate titles, pages that return 404 from an internal link. A crawl framework assumes the opposite. It expects to be throttled, blocked and served errors, and most of its code is about recovering from that politely.

Picking from the wrong half is the common mistake. A desktop auditor pointed at a large third-party site produces a slow crawl and an angry server administrator. A crawl framework pointed at your own 800-page site is a week of work to replace a tool that would have finished in four minutes.

The three things that quietly shrink a crawl

Links that only exist after JavaScript runs. A plain fetcher sees the shell, finds no hrefs, and reports a five-page site. Fetch one page with and without rendering and compare, which is what the rendering comparison covers.

Rules the crawler honors. A robots.txt group matched more broadly than you expected, or an x-robots-tag header on a whole directory, removes pages from the crawl without any error to notice.

Errors counted as endings. A rate limited crawl starts collecting 429 responses, and a crawler that treats those as dead URLs rather than as retries reports a site far smaller than it is. Always read the status code distribution before you trust the page count.

Related

Scraping APIs covers hosted services that do the fetching for you, and scraping routers covers the ones that choose which backend does it per request. Search APIs covers querying an index somebody else built rather than building one. Open source projects lists the crawl and extraction libraries with their licenses. Browser APIs covers the rendering layer these frameworks call out to.