Search APIs, and what each index actually covers

Fourteen ways to query the web programmatically. They divide into three groups that answer different questions, and mixing them up is the common and expensive mistake.

• Google Custom Search JSON API

  Free tier, then paid

  Google's own JSON API for querying a Programmable Search Engine, returning web and image results in the OpenSearch format.

  Suits

  Google results from Google, with 100 free queries a day and no scraping involved, which is the only arrangement here with no terms-of-service question attached.

  Does not suit

  Google's documentation states the API is closed to new customers, and existing customers have until 1 January 2027 to move to something else. It was also designed to search a configured list of sites rather than the open web, so it was never a general search API even while it was open.

• Brave Search API

  Free tier, then paid

  Programmatic access to Brave's own web index, with result metadata and summarized answers as separate plans.

  Suits

  An independent index rather than a reseller of someone else's, so results are not a proxy for Google and the terms permit the use directly. The published rate is $5 per 1,000 requests with $5 of free credit each month and 50 queries per second.

  Does not suit

  An independent index is a different index. Coverage of long-tail and very recent pages differs from Google's, so it cannot answer questions about Google rankings, and anything benchmarked against Google results will show a gap that is real rather than a bug.

• SerpApi

  Paid

  A scraping API returning parsed results pages from Google, Bing, Baidu, YouTube, Amazon, app stores and others.

  Suits

  Breadth of engines and result types, with parsed structures for knowledge panels, local packs and shopping results that would each be a separate parser if you built it.

  Does not suit

  You are buying scraped results, so the legal posture is the vendor's rather than the engine's, and any layout change upstream can move a field you depend on. Priced per search, which makes bulk rank tracking across many keywords and locations add up quickly.

• Serper

  Free tier, then paid

  A low-cost Google search API returning results for web, images, news, maps, places, videos, shopping and scholar queries.

  Suits

  Fast responses at a low per-query price, with 2,500 free queries and no card required, which makes it practical to prototype something query-heavy before committing.

  Does not suit

  Google only. If your question involves Bing, Baidu or a marketplace, this is not the tool, and the same scraped-results caveat applies as to every Google API that is not Google's.

• SearchApi

  Paid

  A real-time SERP API covering Google and other engines, with structured output per result type.

  Suits

  Location and device parameters are explicit per request, which matters because a rank measured without pinning both is not a rank anyone can reproduce.

  Does not suit

  Same structural exposure as the rest of the SERP group: the product is a parser over someone else's page, so an upstream redesign is an outage you did not cause and cannot fix.

• DataForSEO

  Paid

  A suite of SEO data APIs covering SERPs, keyword volumes, backlinks, on-page checks and business listings.

  Suits

  One vendor and one billing relationship for data that otherwise comes from four, with a queued task model that suits large scheduled jobs better than a synchronous API does.

  Does not suit

  The queued model means results arrive later rather than now, so it is a poor fit for anything interactive. The API surface is large and inconsistently shaped across products, and the learning curve is real.

• Oxylabs SERP Scraper API

  Paid

  A SERP scraping API with location targeting, built on the same proxy network as the company's other products.

  Suits

  Location targeting down to city level, which is the whole job for local rank tracking where national results are the wrong answer.

  Does not suit

  Priced and contracted for continuous enterprise volume rather than occasional queries, and it comes with the same account review as the rest of the vendor's catalog. A small project should not start here.

• Exa

  Free tier, then paid

  A search API built on its own neural index, which matches on meaning and can return page contents alongside results.

  Suits

  Describing the kind of page you want works where keyword matching does not, and returning cleaned page content with the result removes a fetch and parse step from a retrieval pipeline.

  Does not suit

  Semantic matching is the wrong tool for an exact lookup: a specific error string or product code is a keyword query, and a meaning-based index will return things that are merely similar. Its index is also its own, so it says nothing about Google.

• Tavily

  Free tier, then paid

  A search and extraction API built for AI agents, returning ranked results with content already cleaned for prompt use.

  Suits

  The output is shaped for a model rather than a person, so there is no boilerplate stripping or token budgeting step between the API and the prompt.

  Does not suit

  That shaping is opinionated and lossy. You get the vendor's idea of the relevant extract, not the page, so anything needing the full document or its markup requires a second fetch anyway.

• spider.cloud Search API

  Free tier, then paid

  A search endpoint that returns result titles, URLs and snippets, and fetches and cleans each result page in the same call when fetch_page_content is set.

  Suits

  Search and retrieval are one request. Result pages come back through the same fetch path as the crawl endpoint, proxies, browser rendering and Markdown conversion included, so a query that needs the text of its top five results is one call instead of a search plus five fetches. Location, country, language and a cap on results per site are request parameters.

  Does not suit

  The documentation never names the index behind the results, so you cannot attribute a position here to any engine, which rules it out for rank measurement. Turning on page fetching makes a query a small crawl, and billing meters bandwidth and compute in credits rather than charging per search, so what a query costs is something you measure rather than read off a price page.

• Kagi Search API

  Paid

  Programmatic access to Kagi's search results, inheriting the account's own settings such as blocked and promoted domains.

  Suits

  Results inherit your account's personalization rules, so a domain you have downranked in the product stays downranked through the API. Ad-free by construction, with a privacy posture that is the company's stated product.

  Does not suit

  A small index and a paid-account model, so this is a tool for a person's own searching rather than an infrastructure component. It is also priced well above the volume APIs, which rules out anything running thousands of queries an hour.

• Perplexity Sonar API

  Paid

  An API that answers a question in prose with citations, running its own search and synthesis rather than returning a result list.

  Suits

  A cited answer in one call, which collapses search, fetch, read and summarize into a single request for question-shaped work.

  Does not suit

  You cannot see the ranking, so it cannot be used for measurement of any kind, and the answer is a model's summary of pages rather than the pages. When the summary is wrong the citation is still there, which makes errors harder to spot rather than easier.

• You.com API

  Paid

  Search and news APIs plus a research endpoint, aimed at grounding language model applications in current web results.

  Suits

  Result snippets are returned at a length tuned for prompt context, and the research endpoint runs a multi-step search for questions that need more than one query.

  Does not suit

  A smaller index than the incumbents and less public detail about coverage, so evaluation means running your own queries and comparing. Built for grounding rather than for measurement, so it will not answer a ranking question.

• Mojeek Web Search API

  Paid

  An API over Mojeek's independently crawled index, one of the few not derived from Google or Bing.

  Suits

  A genuinely independent crawl with no tracking and no personalization, which makes results reproducible across time and location in a way the big engines are not.

  Does not suit

  The index is much smaller than the major engines, so recall on niche or very recent queries is noticeably lower. That is a fair trade for research on ranking bias and a bad one for finding everything on a topic.

Three products, one word

Rank measurement. You want to know what position a URL holds for a query in a place, and the only honest source is the results page itself. That means a SERP API, and it means pinning location, language and device on every request, because a rank without those is a number that cannot be reproduced.

Retrieval. You want documents that answer a question, and you do not care where they rank. Independent indexes and the neural search APIs are built for this, they cost far less per query than SERP scraping, and their results being unlike Google's is not a defect.

Answers. You want prose with citations. Useful when a person reads the output, and unusable as a measurement, because the ranking that produced the answer is not visible.

Paying SERP prices for retrieval is the expensive version of this mistake. Using a retrieval API for rank tracking is the version that produces confident, wrong numbers.

The index is the product

Every difference you will notice comes from the index behind the API: when it crawled, what it reached, and what it chose not to keep. Two APIs disagreeing on a query is usually not a bug in either.

Crawl date is the part people forget. An index that last fetched a page in March answers with March's content, and a page that has since started returning 403 to crawlers may still be in that index looking healthy. If your own pages are missing from an index, the reason is often on your side and mechanical: a robots.txt rule, an x-robots-tag header, a canonical URL pointing somewhere else, or content that only exists after JavaScript runs. Fetching the page yourself with that crawler's agent string tells you which of those it is, and how a crawler resolves a page sets out the order those steps run in.

What AI crawlers have changed

The search APIs built for language models are one half of a shift. The other half is the crawlers that feed those models, which visit sites directly under their own names. GPTBot, ClaudeBot, PerplexityBot and CCBot each have a documented user agent and a robots token, which means a site can allow or deny each one deliberately.

Worth knowing before you set those rules: blocking a crawler and staying out of the answers it produces are not the same thing. Several of these products also retrieve pages live at query time under a different agent, and some answer from an index they built earlier. The Common Crawl archive, for instance, keeps serving pages long after a site has blocked the crawler that collected them.

Related

Crawlers covers software that builds an index rather than queries one. Scraping APIs covers fetching arbitrary pages. Proxy providers covers the network layer under the SERP group.