Open source crawling and extraction projects

Every license here was read from the project's own repository rather than from a directory. Two of these are copyleft in a way that reaches your code, and those rows say so.

• Scrapy

  Open source · BSD-3-Clause · Python

  A Python crawling framework with its own scheduler, middleware chain, item pipelines and throttling.

  Suits

  Concurrency, retries, duplicate filtering and export pipelines are already solved, with a settings system that makes rate limits and politeness configuration rather than code.

  Does not suit

  No browser, so JavaScript-dependent pages need a separate rendering integration that gives back most of the throughput. Its async model is built on Twisted rather than asyncio, which is an unfamiliar shape for anyone who learned Python recently.

• Crawlee

  Open source · Apache-2.0 · TypeScript

  A crawling library for Node and Python covering request queues, retries, proxy rotation, session handling and both HTTP and browser crawlers.

  Suits

  One interface over plain fetching and full browser automation, so testing whether a target needs rendering is a change of crawler class rather than a rewrite.

  Does not suit

  It owns storage and queueing in its own layout, so slotting it into a pipeline that already has both means fighting the framework. The browser crawlers inherit every memory cost of the browser underneath.

• Colly

  Open source · Apache-2.0 · Go

  A Go scraping framework with a callback-driven API, parallelism controls and built-in caching.

  Suits

  Very fast and very light on memory compared with the Python and Node frameworks, and it compiles to a single binary with no runtime to install on the host.

  Does not suit

  No JavaScript execution at all, and the callback style scatters crawl logic across handlers in a way that gets hard to follow past a few hundred lines. Development has been quiet, so expect to read the source rather than a changelog.

• spider

  Open source · MIT · Rust

  A Rust crawling and scraping crate covering the fetch loop, concurrency limits, link discovery and content conversion.

  Suits

  MIT licensed, the permissive end of this list, and cargo features let a build leave out the browser, the cache and the model integrations it does not use.

  Does not suit

  It crawls and converts, so deciding which part of a page is the content is still your job. Nearly all of the work comes from one maintainer, which is a real availability risk for anything you build on top of it.

• spider-py

  Open source · MIT · Rust with a Python interface

  Python bindings for the spider crate, installed as spider_rs, exposing a Website object that crawls, scrapes and hands back links and pages.

  Suits

  The crawl loop runs in Rust, so concurrency and the fetch path are not Python's problem, and per-path budgets and blacklist patterns are set on the builder rather than written as middleware. Its own benchmark reports 150,387 pages in 186 seconds where Scrapy reached 49,598 in an hour on the same site, which is the maintainer's number on the maintainer's laptop and worth re-running on yours.

  Does not suit

  The binding exposes a fraction of what the crate does, so anything missing means writing Rust. It is a compiled extension, so a platform with no published wheel builds it through maturin. The repository has gone eight months without a commit while the crate underneath keeps releasing, so the Python side lags what the Rust side does.

• spider-nodejs

  Open source · MIT · Rust with a Node interface

  Node bindings for the same crate, published as @spider-rs/spider-rs, with a per-page callback, per-path budgets and an optional headless Chrome mode.

  Suits

  Crawling happens on Rust threads instead of the event loop, and a crawl can be handed to a background thread and awaited later, so the process is not parked while pages come in.

  Does not suit

  Same binding gap as the Python port, and a native module means a prebuilt binary per platform, which containers on less common architectures will find out about at install time. Crawlee does the same job in TypeScript with far more documentation, so the argument for this one is throughput and nothing else.

• Katana

  Open source · MIT · Go

  A fast crawler for URL discovery, with a headless mode and parsing of endpoints found inside JavaScript files.

  Suits

  Finds endpoints that link-following misses, because it reads JavaScript bundles for URLs rather than only following anchors.

  Does not suit

  It enumerates URLs and stops there: no extraction, no storage model worth the name. Defaults are tuned for security reconnaissance rather than politeness, so pointing it at a site you do not own needs the rate settings changed first.

• Heritrix

  Open source · Apache-2.0 · Java

  The Internet Archive's archival crawler, writing WARC files that preserve requests and responses byte for byte.

  Suits

  Archival fidelity that nothing else on this list attempts, with output that replays later exactly as it was served, headers included.

  Does not suit

  Fidelity is the entire design, so extraction work fits badly: you get bytes, not fields. Configuration is XML-heavy and the operational learning curve is steep enough that WARC output needs to be a requirement, not a preference.

• Crawl4AI

  Open source · Apache-2.0 · Python

  An async Python crawler that renders pages with a browser and returns Markdown and structured output shaped for language models.

  Suits

  Rendering, content cleaning and Markdown conversion in one library, with extraction strategies that can be rules or a model call depending on how regular the page is.

  Does not suit

  It moves fast and the API has changed between releases, so pin the version and read the changelog before upgrading. Every crawl carries a browser, which makes it heavy for targets that never needed one.

• Firecrawl

  Open source · AGPL-3.0 · TypeScript

  A crawling and scraping service, also published as source, that converts pages to clean Markdown for model and retrieval pipelines.

  Suits

  The hosted product and the source are the same system, so a prototype can move in-house without a rewrite, and the Markdown output removes a cleaning step downstream.

  Does not suit

  The repository is AGPL-3.0, which is a real constraint for anyone embedding it in a service rather than running it as a separate tool, and is the reason a commercial license exists. Markdown conversion also drops attributes and table structure that some extractions need.

• Trafilatura

  Open source · Apache-2.0 · Python

  A Python library that extracts main text, comments and metadata from web pages, with output as text, XML or JSON.

  Suits

  It performs well in published comparisons of boilerplate removal, handles date and author extraction, and runs fast enough for corpus-scale work without a browser.

  Does not suit

  Built for article-shaped pages. A product grid, a dashboard or a documentation page with heavy navigation confuses the main-content heuristic, and there is no way to tell it which region you meant.

• Beautiful Soup

  Open source · MIT · Python

  A Python library for navigating and searching parse trees, sitting on top of a parser you choose.

  Suits

  Forgiving with broken markup and readable enough that a selector written a year ago still explains itself. The documentation is among the best in Python.

  Does not suit

  Slow. On a corpus of hundreds of thousands of pages the parse step becomes the bottleneck, and the usual fix is calling lxml directly. It is also only a tree API, so it decides nothing about which part of the tree is content.

• lxml

  Open source · BSD-3-Clause · Python

  Python bindings for libxml2 and libxslt, providing fast XML and HTML parsing with XPath and XSLT support.

  Suits

  Much faster than pure-Python parsers, with full XPath, which is the right tool when a selector has to express a structural condition CSS cannot.

  Does not suit

  It is a C extension, so installation can fail on platforms without wheels and it complicates minimal containers. XPath is also more power than most extraction needs, and unreadable XPath is a maintenance cost that shows up later.

• Cheerio

  Open source · MIT · TypeScript

  A server-side implementation of a jQuery-like API over a parsed HTML tree, for Node.

  Suits

  Familiar selector syntax and no browser, which makes it many times faster and lighter than parsing with a headless browser for markup that is already in the response.

  Does not suit

  It parses markup and nothing else: no layout, no JavaScript, no computed styles. Any check that depends on whether an element is visible has to happen in a real browser instead.

• Readability

  Open source · Apache-2.0 · JavaScript

  The content extraction library behind Firefox Reader View, isolating the main article from a page.

  Suits

  Battle-tested across the open web at browser scale, and the same algorithm users already see, which makes its output predictable to anyone who has used Reader View.

  Does not suit

  Article-shaped pages only, and it runs against a DOM, so a Node deployment needs jsdom or similar alongside it. It discards structure you may have wanted, including most attributes.

• fast_html2md

  Open source · MIT · Rust

  A Rust library that turns HTML into Markdown, with a lol_html streaming rewriter as the default path and a scraper-based parser behind a feature flag.

  Suits

  Conversion as its own library rather than a step buried in a crawler, so a pipeline that already fetches pages gets Markdown output without adopting a framework. A streaming mode returns chunks as they convert, so a large document never has to sit in memory as one string.

  Does not suit

  It converts, it does not extract. Hand it a whole page and the navigation, the footer and the cookie banner come back as Markdown too, so an extractor still has to run first. Conversion also drops attributes and most table structure, which is the trade every converter here makes.

• Docling

  Open source · MIT · Python

  A document conversion toolkit that turns PDF, DOCX, PPTX, HTML and images into structured Markdown or JSON, with layout and table recognition.

  Suits

  Table structure and reading order survive conversion, which is where naive PDF text extraction fails and quietly produces scrambled rows.

  Does not suit

  Layout and table models mean real compute: CPU-bound on a laptop and slow enough that a large batch wants a GPU. It is a document pipeline, not a crawler, so fetching is still yours.

• Requests

  Open source · Apache-2.0 · Python

  The standard Python HTTP client, handling sessions, cookies, redirects and authentication.

  Suits

  So widely used that almost every example you find already assumes it, and its session object handles cookie and connection reuse correctly without configuration.

  Does not suit

  Synchronous, so concurrency means threads or a different library, and its TLS fingerprint is that of a Python client rather than a browser, which some bot mitigation matches on directly. Neither is a bug, but both decide whether it fits.

• chromey

  Open source · Apache-2.0 · Rust

  A Rust library that drives Chrome over the DevTools Protocol, importing as chromiumoxide, with feature flags for adblocking, request filtering, response caching and downloading Chrome for Testing.

  Suits

  Apache-2.0 rather than nodriver's AGPL-3.0 two rows down, which is the difference that decides whether a crawler can embed it, and blocking ads and trackers is a cargo feature rather than an extension you load. It can fetch its own Chrome build, so a container does not need one installed.

  Does not suit

  Chrome only, and you are at protocol level: no locator API, no auto-waiting, no test runner, so every wait and retry is yours to write. The crate was renamed but still imports as chromiumoxide, which makes searching for an answer confusing, and the documentation is the README and the examples.

• undetected-chromedriver

  Open source · GPL-3.0 · Python

  A patched ChromeDriver for Selenium that removes several properties commonly used to identify automated Chrome.

  Suits

  A drop-in replacement for the standard driver in existing Selenium code, which makes testing whether a target checks those properties nearly free.

  Does not suit

  Licensed GPL-3.0, which is stricter than most of this list and matters if you distribute your crawler. It also chases a moving target: each Chrome release can undo the patch, and using it to reach content a site has refused to automated clients is a terms question, not just a technical one.

• nodriver

  Open source · AGPL-3.0 · Python

  An async Python browser automation library that speaks the DevTools Protocol directly, with no webdriver binary involved.

  Suits

  Removing the driver process removes both a startup cost and one of the signals detection looks for, and the async API fits modern Python without a compatibility layer.

  Does not suit

  AGPL-3.0, which is the most demanding license here and reaches services offered over a network, so read it before embedding. Chrome only, younger than the alternatives, and the documentation assumes you already know the protocol.

• Camoufox

  Open source · MPL-2.0 · C++ with a Python interface

  A Firefox build patched at the C++ level so that fingerprint properties are changed below the JavaScript layer, with a Python interface.

  Suits

  Changing fingerprint values in the engine rather than by injecting scripts avoids the inconsistencies that script-level spoofing leaves behind and that detection specifically looks for.

  Does not suit

  A custom browser build is a custom browser build: large downloads, its own release cadence, and a lag behind upstream Firefox security fixes. Licensed MPL-2.0, and the browser it patches carries its own terms.

How the licenses here were checked

Each license came from the project's own repository metadata or its license file, not from a package directory or an aggregator. That distinction earns its keep: a repository can carry a license file the package metadata never reflects, and directory sites copy each other's mistakes.

Two entries deserve a second look before you build on them. Firecrawl and nodriver are AGPL-3.0, whose network clause reaches software offered as a service, which is exactly what a crawler usually is. undetected-chromedriver is GPL-3.0. Where the boundary falls depends on how you combine the code, and that is a question for your lawyer rather than for a listing page. The point of naming it here is that people usually find out later.

Free software, paid operations

Nothing on this list charges for a download, and none of it is free to run. The recurring costs are the same four every time: exit addresses when a target blocks by network, memory when a page needs a browser, retry logic when a site is slow or flaky, and somebody's afternoon when a selector stops matching.

A hosted service prices all four into one number. Self-hosting unbundles them, which is a win when you have engineers and a loss when you have a deadline. The honest comparison is not the license fee against the subscription, it is the subscription against a named person's time.

Pick a parser, then pick an extractor

These are separate jobs and the split is worth keeping. A parser turns bytes into a queryable tree, which is lxml, Cheerio or Beautiful Soup. An extractor decides which part of the tree is the content, which is Trafilatura or Readability.

Skipping the second step is the usual mistake. A hand-written rule that grabs the largest div works on the site you tested it against and produces navigation menus everywhere else. If your pages are articles, use an extractor built for articles. If they are product pages, accept that you are writing per-site selectors and plan for the maintenance rather than pretending a general rule exists.

Before you blame the library

When a crawl comes back empty, the library is rarely the problem. Check what the server actually returned first. A 403 means the request was refused, a 429 means you were too fast, and a page that returns 200 with a few hundred bytes is usually a challenge page being stored as content. A robots.txt rule or a noindex directive changes what you should be requesting at all.

The rendering comparison covers what survives without JavaScript, and how a crawler resolves a page sets out the order these steps run in.

Related

Crawlers sets these frameworks beside the commercial site auditors. Browser APIs covers the rendering layer. Scraping APIs covers the hosted alternative when maintaining this yourself stops being worth it. Browser agents covers the model-driven layer above these, where an AGPL and an SSPL entry raise the same license question as the copyleft rows here. Search APIs covers querying an index somebody else maintains instead of crawling for one.