common crawl explained: URL index API and web graph

Common Crawl publishes several datasets for different jobs, and the names are easy to mix up. This guide separates archive captures from hyperlink-graph edges, shows a small URL Index request and safe pagination, and explains which limitations survive every method. It also shows where the open graph fits among the practical options for free backlink research.

what common crawl actually is

Common Crawl is a nonprofit that has collected web data since 2008. Its archive is a sample of the web, not a complete inventory. A crawl collection and a hyperlink-graph release are different artifacts: finding a URL in one does not establish that its links appear in a particular graph. Start with the question you need to answer.

dataset	what it contains	use it for
WARC	Archived responses and associated records	Inspecting captured page content
WAT / WET	Extracted metadata / plain text	Processing metadata or text without parsing full responses
URL index	Capture locations and metadata	Finding whether and where a URL was archived
Web graph	Host or domain nodes and directed edges	Researching relationships between sites

The file-format guide explains WARC, WAT and WET. The archive formats differ from the Parquet columnar URL index, a separate access path for large analytical queries. For a small lookup, the HTTP index service avoids downloading an entire collection.

how to query the common crawl URL index API

Use the index server to choose an archive. Its collection list exposes each collection's id and cdx-api endpoint:

bash

curl --fail --silent --show-error https://index.commoncrawl.org/collinfo.json

Copy an endpoint from that list. Do not invent a collection name from today's month or substitute a quarterly graph identifier. The example below pins CC-MAIN-2026-34; choose another listed collection when you need another capture period. This is a URL-index collection, not a promise about crawlgraph's active graph.

bash

CC_INDEX='https://index.commoncrawl.org/CC-MAIN-2026-34-index'
curl --fail --get "$CC_INDEX" \
  --data-urlencode 'url=example.com/*' \
  --data-urlencode 'output=json' \
  --data-urlencode 'filter==status:200' \
  --data-urlencode 'limit=5'

This asks for up to five successful-response captures under the URL prefix. Output is newline-delimited JSON: parse each line separately. A capture can include a timestamp and the archive filename, offset and length. Those locate stored content; they are not a list of sites linking to that URL.

a sample is not a full export

Five records only prove that those captures were returned. They do not measure the site's indexed pages, referring domains or current backlinks. For graph extraction, use the separate Common Crawl web graph query tutorial.

pagination: count blocks, then request a page

The CDX API reference documents showNumPages=true, which returns pagination metadata. Request page zero first and stop before the reported page count. Keep the collection, URL pattern and pageSize identical between counting and retrieval.

bash

curl --fail --get "$CC_INDEX" \
  --data-urlencode 'url=example.com/*' \
  --data-urlencode 'showNumPages=true' \
  --data-urlencode 'pageSize=1'

# Pause before the next request.
sleep 3
curl --fail --get "$CC_INDEX" \
  --data-urlencode 'url=example.com/*' \
  --data-urlencode 'output=json' \
  --data-urlencode 'pageSize=1' \
  --data-urlencode 'page=0'

Here pageSize=1 means one compressed index block, not one capture. A page can therefore contain many rows; neither its page count nor its block count is a URL total. The second command fetches only page zero, not all pages. Do not keep the earlier five-row limit when you intend to retrieve each complete page.

In our September 11, 2026 check, collection discovery, the five-record sample and the page-count request returned HTTP 200. Page-zero retrieval returned HTTP 504 twice. The pagination syntax follows the API documentation, but a valid request can still time out. Save successful output, stop on repeated failures and resume later rather than assuming an empty result.

query the public service carefully

Pause between requests and avoid parallel queries. If the service returns 503, stop and back off rather than retrying through proxies. Common Crawl's FAQ explains temporary blocking and recovery. For broad filtering or aggregation, use its columnar index instead of treating the public API as a bulk download service. A three-second pause is an example, not a guaranteed safe rate.

what the web graph numbers mean

The Apr-Jun 2026 release in our shared manifest reports 121.1M domains and 3.90B domain-level edges. An edge connects a source and destination; it does not count every individual linking page. See referring domains versus backlinks before comparing totals.

Common Crawl publishes host and domain graphs with different aggregation. Graph links can include technical relationships, not just editorial recommendations. A large edge count therefore does not establish link quality or explain why Google ranks a site.

how crawlgraph uses the data

crawlgraph makes referring-host lookups from Common Crawl graph data available through a web interface and a backlink API. That product API is separate from the public URL Index API above. Large lookups return at most 100,000 referring-host rows alongside the total; an export is not necessarily every matching row for a very large site. Release availability is documented in our Common Crawl release guide.

what common crawl cannot tell you

An absent URL or edge does not prove that a page or link never existed. Coverage, crawl timing and graph aggregation all affect the result. The crawler does not execute JavaScript, so links that require client-side rendering may be missing. Neither an archived capture nor a graph edge proves that a link is still live today.

Check candidate source pages before outreach and record the collection or graph release used in an analysis. Compare like-for-like units and dates across tools. We do not claim a measured overlap with a commercial index, or that every SEO audit can use this dataset in place of a fresh crawl.