the link data behind this product is not proprietary. common crawl publishes its domain-level hyperlink graph as three gzipped text files per release, free, with no api key and no signup, and pulling one domain's referring domains out of them is a streaming scan that finishes on a laptop. this is how to do it. if you want the concepts first - nodes, edges, the host roll-up, how authority scores fall out - what is a backlink graph is the theory and this is the implementation.
it is a fair question, since this is our product. the answer is that the reproducibility is the argument. a number from a proprietary index has to be taken on trust. a number from this graph can be checked by anyone willing to spend an afternoon, and we would rather be the tool that saves you the afternoon than the tool whose claims cannot be audited.
the three files
# one release, three files. paths follow the release id. <release>-domain-vertices.txt.gz ~200 MB id -> reversed domain <release>-domain-edges.txt.gz ~16 GB from_id to_id <release>-domain-ranks.txt.gz ~2.4 GB harmonic + pagerank values # vertices: note the domain is REVERSED, for sort locality 1042 com.example 3 # id rev_domain num_hosts # edges: numeric ids only. this is why you need the vertex file. 1042 880134
two details drive every design decision downstream. first, the edge list holds numeric ids only, so you cannot do anything useful without the vertex file to translate them. second, domains are stored reversed, com.example rather than example.com, so that sorting groups all hosts of a domain and all domains of a tld together. reverse it back before showing it to a person.
the pipeline
the edge list does not fit in memory and does not need to. for a single domain the whole job is three passes, and only a small set of ids is ever held:
#!/usr/bin/env python3
"""Backlinks for one domain from the Common Crawl domain graph.
Two streaming passes. Nothing is ever fully loaded: the edge list does
not fit in memory on a normal machine and does not need to.
"""
import gzip, sys
TARGET = "com.example" # reversed, as stored in the vertex file
# pass 1: resolve the target domain to its numeric id.
# one linear scan; stop as soon as it is found.
target_id = None
with gzip.open("vertices.txt.gz", "rt") as fh:
for line in fh:
vid, rev_domain, _ = line.rstrip("\n").split("\t", 2)
if rev_domain == TARGET:
target_id = vid
break
if target_id is None:
sys.exit(f"{TARGET} not in this release")
# pass 2: stream the edge list, keep only edges pointing AT the target.
# the set of source ids is small - that is the only thing held in memory.
linking_ids = set()
with gzip.open("edges.txt.gz", "rt") as fh:
for line in fh:
src, dst = line.rstrip("\n").split("\t", 1)
if dst == target_id:
linking_ids.add(src)
# pass 3: resolve those ids back to names. one more scan of the small file.
with gzip.open("vertices.txt.gz", "rt") as fh:
for line in fh:
vid, rev_domain, _ = line.rstrip("\n").split("\t", 2)
if vid in linking_ids:
# un-reverse for humans: com.example -> example.com
print(".".join(reversed(rev_domain.split("."))))that is the complete method. the run time is dominated by decompressing 16 GB once, so expect minutes rather than seconds, and expect it to be i/o bound rather than cpu bound. if you are doing this repeatedly, the obvious optimisation is to extract the id-to-name mapping once and keep it, rather than rescanning the vertex file each run.
building a full {id: domain} dictionary from the vertex file before touching the edges. it works, and it costs gigabytes of ram for a lookup you only need for a few thousand ids. collect the ids first, resolve them after. that single reordering is the difference between a laptop job and a swap-thrashing one.
adding authority
the ranks file carries common crawl's published harmonic centrality and pagerank values per domain, so you do not need to compute centrality yourself, and you should not want to: that is the computation that genuinely requires a cluster. joining the ranks file onto your list of linking ids gives you a sorted list, strongest linker first.
the exact normalisation we apply on top of harmonic rank to produce a 0-100 score is written out in what is a backlink graph, and the comparison against the vendor metrics is in domain authority vs domain rating vs trust flow.
what this will not give you
being straight about the ceiling, because it decides whether this is the right layer for your problem:
- no anchor text and no rel attributes. this is a domain-level graph. anchor text and
nofollowlive in the html of individual pages and do not survive the roll-up. - no page-level links. you get
news.org links to shop.com, not which article linked to which url. - snapshot freshness. releases are quarterly, so a link created last week is not in it. a continuous commercial crawl wins on recency and always will.
for competitor link discovery none of those limits bite, because the question is which domains link to a rival and not to you. for anchor-text auditing they are fatal, and a page-level commercial index is the correct tool.
or skip the afternoon
everything above is what our api does for you, against an index we already built and refresh each release. the free tier is 15 backlink lookups a month with a self-serve key and no card, documented at the api docs, and there is an mcp server so an assistant can call it directly. recipes are in programmatic backlink research.
faq
Can I query the Common Crawl web graph myself?
Yes. The domain-level web graph is published as three gzipped text files per release: a vertex list mapping numeric ids to reversed domain names, an edge list of id pairs, and a ranks file with harmonic centrality and PageRank values. They are free to download and carry no API key, and the whole pipeline for extracting one domain's backlinks is a streaming scan that runs on an ordinary laptop.
How big are the Common Crawl web graph files?
For a recent release the vertex file is roughly 200 MB gzipped, the ranks file around 2.4 GB gzipped, and the edge list around 16 GB gzipped. The edge list is the one that dictates your approach: it does not fit in memory on a normal machine, so you stream it rather than loading it.
Do I need Spark or a cluster to process it?
Not for a single domain's backlinks. That job is two streaming passes over the edge list with a small set of ids held in memory, which finishes on a laptop. You need a cluster when you want global computations across all domains at once, such as recomputing centrality yourself, rather than answering a question about one node.
Why are domains reversed in the vertex file?
The vertex file stores com.example rather than example.com so that sorting the file groups every host under the same domain together, and every domain under the same top-level domain together. It is the same reason Java package names are written that way. Reverse it back before displaying anything to a human.
How is this different from a commercial backlink API?
Coverage and freshness differ, and so does verifiability. A commercial index runs a continuous crawl and will see brand-new links sooner. The open graph is a quarterly snapshot, but it is public, so any number derived from it can be reproduced independently by anyone willing to process the files, which is not true of a number from a proprietary index.
writes the queries we run internally. ships one tactical post a week.
plus one when a new common crawl release lands. that is all.