backlink API for referring domains and link gaps

current crawlgraph product facts

crawlgraph is a focused backlink and competitor-gap tool built on Common Crawl snapshots. It does not run a live daily crawler, and a Common Crawl graph cannot contain every backlink on the live web.

1. Quickstart

Start with two requests: ask for a free key, then use the key from your email to look up the referring domains for any site. No account or card is required.

# 1. Request a key (it arrives by email)
curl -X POST https://crawlgraph.com/api/v1/free-key \
  -H "Content-Type: application/json" \
  -d '{"email": "[email protected]"}'

# 2. Run your first useful query
curl -X POST https://crawlgraph.com/api/v1/backlinks \
  -H "Authorization: Bearer cg_live_…" \
  -H "Content-Type: application/json" \
  -d '{"domain": "example.com", "limit": 10}'

Response (excerpt)

{
  "domain": "example.com",
  "release_id": "cc-main-2026-apr-may-jun",
  "release_label": "Apr-Jun 2026",
  "total_linking_domains": 4821,
  "returned": 10,
  "results": [
    { "linking_domain": "blog.foo.com", "num_hosts": 12, "tld": "com",
      "cg_authority": 84, "cg_rank": 1421 },
    { "linking_domain": "news.bar.org", "num_hosts": 7,  "tld": "org",
      "cg_authority": 71, "cg_rank": 9402 }
  ]
}

What to try first

• Look up your own domain to see which referring sites rank first.
• Repeat the request with one competitor's domain.
• Save both JSON responses and compare their linking_domain lists manually.
• Connect the hosted MCP server with the same key and run backlink lookups from your AI client.

Still need the key for step 2? Request it here without leaving the quickstart.

2. Authentication

Every request to /api/v1/* needs a bearer token in the Authorization header:

Authorization: Bearer cg_live_<your-key>

• Keys are prefixed with cg_live_ and roughly 52 characters long.
• Get a key → from your account page. You can have up to 10 active keys per user and label each one (e.g. production, n8n-bot).
• The full key is shown only once at creation. If you lose it, revoke and create a new one - there's no recovery path.
• Free keys can call backlinks. Gap analysis and outreach targets require the lifetime tier.

3. MCP server

The hosted https://crawlgraph.com/mcp endpoint is the zero-install way to connect an AI client over Model Context Protocol without running a local process. It uses the same bearer key and quotas as the HTTP API. The open-source crawlgraph-mcp npm package remains available as a local fallback. The current local package documents five read-only tools; hosted package versions are released separately and must be checked with tools/list before relying on a newly added tool.

▶ watch the 30-second mcp setup video

Hosted setup: Codex CLI

Codex reads the token from your environment, so it does not need to be copied into the MCP config:

export CRAWLGRAPH_API_KEY="cg_live_<your-key>"
codex mcp add crawlgraph \
  --url https://crawlgraph.com/mcp \
  --bearer-token-env-var CRAWLGRAPH_API_KEY

Hosted setup: Claude Code

Claude Code accepts an HTTP endpoint plus an explicit authorization header. Its local scope keeps the setting private to the current project; do not commit or share the resulting client config:

claude mcp add --scope local --transport http \
  crawlgraph https://crawlgraph.com/mcp \
  --header='Authorization: Bearer cg_live_<your-key>'

These two shapes were checked against the installed Codex and Claude Code CLIs on July 16, 2026. Other clients use different remote-server schemas; follow that client's documentation instead of copying one of these shapes blindly.

Local npm fallback

If your client only supports local stdio servers, add this mcpServers block to its local MCP configuration:

{
  "mcpServers": {
    "crawlgraph": {
      "command": "npx",
      "args": ["-y", "crawlgraph-mcp"],
      "env": {
        "CRAWLGRAPH_API_KEY": "cg_live_<your-key>"
      }
    }
  }
}

Troubleshooting hosted connections

• 401: the bearer key is missing, invalid, or revoked. Confirm the header is Authorization: Bearer cg_live_<your-key>.
• 405: the client sent a browser-style GET. The endpoint expects MCP Streamable HTTP POST requests.
• 406: the client omitted the MCP Accept header. It must accept application/json, text/event-stream.
• If an old local server keeps launching, remove the stale client entry before adding the hosted endpoint, then restart the client.

Tools

• backlinks - referring domains for a target, with authority scores. Available to free and lifetime keys.
• backlink_changes - additions, observed absences, and authority movement between Common Crawl releases. Omitted release ids use the newest queryable pair and cost one backlinks call. A removed domain means it was not observed in the newer snapshot, not that a live link was proven deleted; when two queryable snapshots do not exist, the tool returns a successful comparison_available: false response.
• gap_analysis - domains linking to your competitors but not to you.
• gap_outreach_targets - the warm-outreach play: the domains that link to all of your competitors but not to you, de-noised (platforms and CDNs filtered) and ranked by authority. The warmest backlink targets you will ever pitch.
• releases - list the Common Crawl snapshots you can query.

Then just ask your assistant

Once it is connected you describe the goal in plain language and the agent runs the tools for you:

"Use gap_outreach_targets for mydomain.com against
competitor-a.com and competitor-b.com, then draft a short
outreach email to each priority target."

Same auth and quotas as the HTTP API - the MCP server is a thin client over the endpoints documented below. Source, issues and the full tool reference live at github.com/pucilpet/crawlgraph-mcp.

4. Quotas & rate limits

• Window is the calendar month in UTC. Hard reset on the 1st at 00:00 UTC - no rollover.
• Only successful (2xx) calls count. Validation errors, auth failures, and quota rejections are free.
• Failed gap jobs do not refund quota in v1. If something looks wrong, email support and quote the request_id.
• A separate IP-based limiter caps bursts at roughly 60 requests per minute on /api/v1/* to protect the backend.

Response headers

Every 2xx response (and 429s) carries these headers so your client can pace itself:

5. Errors

Every non-2xx response uses the same envelope:

{
  "error": "<code>",
  "message": "<human readable>",
  "request_id": "req_a1b2c3d4"
}

6. Endpoints

POST /api/v1/backlinks

Synchronous backlink lookup for a single domain. Counts against the backlinks quota.

Request body

{
  "domain": "example.com",
  "release_id": "cc-main-2026-apr-may-jun",   // optional; default = latest
  "limit": 1000,                      // optional; default 1000, max 10000
  "sort": "authority"                 // optional; "authority" (default) | "hosts"
}

Response

{
  "domain": "example.com",
  "release_id": "cc-main-2026-apr-may-jun",
  "release_label": "Apr-Jun 2026",
  "total_linking_domains": 4821,
  "returned": 1000,
  "results": [
    { "linking_domain": "blog.foo.com", "num_hosts": 12, "tld": "com",
      "cg_authority": 84, "cg_rank": 1421 },
    { "linking_domain": "news.bar.org", "num_hosts":  7, "tld": "org",
      "cg_authority": 71, "cg_rank": 9402 }
  ]
}

Notes

• limit caps at 10,000. The API is for programmatic use, not bulk export - use the dashboard for full datasets.
• Field names match the internal service: linking_domain, num_hosts, tld. No rename layer.
• cg_authority is a 0-100 log-rank percentile derived from Common Crawl's harmonic centrality (higher = more authoritative). cg_rank is the raw PageRank position across the whole graph (1 = top-ranked domain). Both are null for domains that don't appear in the ranks file.
• sort="authority" (default) orders by cg_authority DESC then num_hosts DESC; sort="hosts" preserves the legacy num_hosts DESC order.
• Malformed domain, unknown release_id, or out-of-range limit → 400 validation_error.
• A known release shown by GET /api/v1/releases with available: false → 409 release_unavailable before quota or DuckDB work.

curl

curl -X POST https://crawlgraph.com/api/v1/backlinks \
  -H "Authorization: Bearer cg_live_…" \
  -H "Content-Type: application/json" \
  -d '{"domain": "example.com", "limit": 500}'

Python (requests)

import os, requests

r = requests.post(
    "https://crawlgraph.com/api/v1/backlinks",
    headers={"Authorization": f"Bearer {os.environ['CG_KEY']}"},
    json={"domain": "example.com", "limit": 500},
    timeout=30,
)
r.raise_for_status()
data = r.json()
print(data["total_linking_domains"], "linking domains")

Node (fetch)

const res = await fetch("https://crawlgraph.com/api/v1/backlinks", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${process.env.CG_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({ domain: "example.com", limit: 500 }),
});
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const data = await res.json();
console.log(data.total_linking_domains, "linking domains");

GET /api/v1/releases

List the Common Crawl releases CrawlGraph has indexed. Read-only, not counted against quota - this is the caller learning what to pass.

Response

{
  "releases": [
    { "id": "cc-main-2026-apr-may-jun", "label": "Apr-Jun 2026", "available": true },
    { "id": "cc-main-2026-jan-feb-mar", "label": "Jan-Mar 2026", "available": true }
  ]
}

curl

curl https://crawlgraph.com/api/v1/releases \
  -H "Authorization: Bearer cg_live_…"

Python (requests)

import os, requests

r = requests.get(
    "https://crawlgraph.com/api/v1/releases",
    headers={"Authorization": f"Bearer {os.environ['CG_KEY']}"},
    timeout=30,
)
r.raise_for_status()
for rel in r.json()["releases"]:
    print(rel["id"], rel["label"])

Node (fetch)

const res = await fetch("https://crawlgraph.com/api/v1/releases", {
  headers: { Authorization: `Bearer ${process.env.CG_KEY}` },
});
const { releases } = await res.json();
for (const r of releases) console.log(r.id, r.label);

POST /api/v1/gap-analysis · GET /api/v1/gap-analysis/{job_id}

Async gap-analysis. POST submits a job (counts against the gap quota); GET polls for status and result. Jobs are retained for 7 days.

POST request body

{
  "my_domain": "example.com",
  "competitor_domains": ["a.com", "b.com", "c.com"]   // 1-5 entries
}

POST response (202)

{
  "job_id": "gap_a1b2c3",
  "status": "queued",
  "poll_url": "/api/v1/gap-analysis/gap_a1b2c3"
}

GET response - running

{
  "job_id": "gap_a1b2c3",
  "status": "running",
  "started_at": "2026-04-27T12:34:56Z",
  "progress_pct": 42
}

GET response - completed

{
  "job_id": "gap_a1b2c3",
  "status": "completed",
  "completed_at": "2026-04-27T12:36:18Z",
  "result": {
    "my_domain": "example.com",
    "competitor_domains": ["a.com", "b.com", "c.com"],
    "gaps": [
      { "linking_domain": "x.com", "found_on": ["a.com", "b.com"] }
    ],
    "total_gaps": 1284
  }
}

GET response - failed

{
  "job_id": "gap_a1b2c3",
  "status": "failed",
  "error": { "code": "internal_error", "message": "..." }
}

Notes

• Max 5 competitors per request - matches the dashboard cap.
• GET returns 404 not_found if the job isn't yours, even if the id exists.
• Failed jobs don't refund quota in v1. Email support with the request_id if it matters.

curl

# 1. Submit
curl -X POST https://crawlgraph.com/api/v1/gap-analysis \
  -H "Authorization: Bearer cg_live_…" \
  -H "Content-Type: application/json" \
  -d '{"my_domain": "example.com", "competitor_domains": ["a.com","b.com"]}'

# 2. Poll
curl https://crawlgraph.com/api/v1/gap-analysis/gap_a1b2c3 \
  -H "Authorization: Bearer cg_live_…"

Python (requests)

import os, time, requests

H = {"Authorization": f"Bearer {os.environ['CG_KEY']}"}
sub = requests.post(
    "https://crawlgraph.com/api/v1/gap-analysis",
    headers=H,
    json={
        "my_domain": "example.com",
        "competitor_domains": ["a.com", "b.com"],
    },
    timeout=30,
).json()

job_id = sub["job_id"]
while True:
    j = requests.get(
        f"https://crawlgraph.com/api/v1/gap-analysis/{job_id}",
        headers=H, timeout=30,
    ).json()
    if j["status"] in ("completed", "failed"):
        break
    time.sleep(5)
print(j)

Node (fetch)

const H = { Authorization: `Bearer ${process.env.CG_KEY}` };

const submit = await fetch("https://crawlgraph.com/api/v1/gap-analysis", {
  method: "POST",
  headers: { ...H, "Content-Type": "application/json" },
  body: JSON.stringify({
    my_domain: "example.com",
    competitor_domains: ["a.com", "b.com"],
  }),
}).then(r => r.json());

const jobId = submit.job_id;
let job;
do {
  await new Promise(r => setTimeout(r, 5000));
  job = await fetch(
    `https://crawlgraph.com/api/v1/gap-analysis/${jobId}`,
    { headers: H },
  ).then(r => r.json());
} while (job.status !== "completed" && job.status !== "failed");
console.log(job);

GET /api/v1/changes

Compares inbound-link observations for one domain across 2 indexed Common Crawl release snapshots. It reports referring domains observed only in the newer snapshot, domains observed in the older snapshot but absent from the newer one, and authority movement. Counts as one call against the backlinks quota - the same bucket as POST /api/v1/backlinks.

This is snapshot comparison, not live link monitoring. Absence from the newer Common Crawl snapshot does not prove that a live page removed a link.

Query parameters

• domain - required; a strict domain such as example.com. Schemes, paths, bare TLDs, and IP addresses are rejected.
• from - optional; older release id. With both release parameters omitted, defaults to the older release in the newest queryable pair.
• to - optional; newer release id. With both release parameters omitted, defaults to the newer release in the newest queryable pair. If to is supplied without from, its nearest queryable ancestor is used.

Response - comparison available

{
  "domain": "example.com",
  "comparison_available": true,
  "from_release": {
    "id": "cc-main-2026-jan-feb-mar",
    "label": "Jan-Mar 2026"
  },
  "to_release": {
    "id": "cc-main-2026-apr-may-jun",
    "label": "Apr-Jun 2026"
  },
  "counts": {
    "from_snapshot": 4821,
    "to_snapshot": 4890,
    "added": 92,
    "removed": 23,
    "authority_moved": 17
  },
  "added": [
    {
      "linking_domain": "new-reference.org",
      "num_hosts": 4,
      "cg_authority": 73
    }
  ],
  "removed": [
    {
      "linking_domain": "older-reference.net",
      "num_hosts": 2,
      "cg_authority": 61
    }
  ],
  "authority_moved": [
    {
      "linking_domain": "example-source.com",
      "from_authority": 52,
      "to_authority": 60,
      "delta": 8
    }
  ],
  "truncated": false,
  "cap": 100000,
  "snapshot_caveat": "Common Crawl snapshots are periodic observations, not live link monitoring. Absence from a newer snapshot does not prove a page removed a link."
}

Response - comparison unavailable

{
  "domain": "example.com",
  "comparison_available": false,
  "from_release": null,
  "to_release": {
    "id": "cc-main-2026-mar-apr-may",
    "label": "Mar-May 2026"
  },
  "counts": {
    "from_snapshot": 0,
    "to_snapshot": 0,
    "added": 0,
    "removed": 0,
    "authority_moved": 0
  },
  "added": [],
  "removed": [],
  "authority_moved": [],
  "truncated": false,
  "cap": 100000,
  "message": "both indexed release artifacts are required for a comparison",
  "snapshot_caveat": "Common Crawl snapshots are periodic observations, not live link monitoring."
}

Notes

• Each snapshot is capped at 100,000 observed referring domains. truncated: true means at least one side exceeded that cap, so counts and arrays describe only the capped comparison.
• comparison_available: false is a successful 200 response when fewer than 2 required release artifacts are queryable. It still consumes one backlinks call; this differs from a single-release 409 release_unavailable.
• Unknown release ids, equal from/to ids, and malformed domains return 400 validation_error without consuming quota.

curl

curl "https://crawlgraph.com/api/v1/changes?domain=example.com" \
  -H "Authorization: Bearer cg_live_…"

7. Webhooks & changelog

Webhooks are coming in a future version. For now, polling the gap-analysis job endpoint is the only async pattern.

The current API is at v1. Breaking changes will land on /api/v2 with a deprecation window - your v1 integrations won't break overnight. Check back here for changelogs.

8. OpenAPI

For tooling integration (Postman, openapi-typescript, Insomnia, etc.) the OpenAPI 3.1 schema is served at /api/v1/openapi.json. It is regenerated on every deploy and provides the machine-readable public routes, parameters, and response models. The examples and snapshot-specific behavioral caveats remain on this page.