programmatic backlink research: 5 api recipes

programmatic backlink research does not need an enterprise API: with one free key, you can pull a domain's referring sites, sort them by authority, compare small competitor sets, and feed the results into an automation or AI client. crawlgraph includes 15 backlink calls each calendar month, with no card. these five copy-and-run recipes turn the broader free backlink research workflow into curl, Node, webhook, and MCP building blocks.

each lookup returns unique referring domains from the Common Crawl web graph, not a dump of every linking page. the result is deliberately small and script-friendly: domain, link count, TLD, authority, and global rank. the full request and error reference lives in the API documentation.

request a free key and set it once

the free tier covers 15 backlink lookups per UTC calendar month. request one key below; it is delivered by email and is never shown in the browser response. there is no card and no account setup to finish before the first call.

if you prefer the command line, the request is one unauthenticated call. once the email arrives, put the key in an environment variable so every later recipe can reuse it:

# request a key — it arrives by email
curl -X POST https://crawlgraph.com/api/v1/free-key \
  -H "Content-Type: application/json" \
  -d '{"email": "[email protected]"}'

# after the email arrives, keep the key in your shell environment
export CG_KEY="cg_live_..."

recipe 1: pull one domain with curl

the core endpoint is synchronous: send a domain and get its referring domains back in the same response. set sort explicitly so anyone reading the script knows that the most authoritative linkers should come first.

curl -sS -X POST https://crawlgraph.com/api/v1/backlinks \
  -H "Authorization: Bearer $CG_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "domain": "example.com",
    "limit": 100,
    "sort": "authority"
  }'

the response includes total_linking_domains, the number of rows returned, and a results array. every result has alinking_domain, num_hosts, tld,cg_authority, and cg_rank. rate-limit headers tell the caller how many of the 15 monthly calls remain, so a script can stop before it hits a 429.

recipe 2: sort and refine by authority

authority sorting already happens on the server, but local filtering lets you reuse one response without spending another call. this example requests 1,000 rows, saves them, removes unranked and lower-authority domains, then keeps the first 25.

# save one API response, then refine it locally without spending another call
curl -sS -X POST https://crawlgraph.com/api/v1/backlinks \
  -H "Authorization: Bearer $CG_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"example.com","limit":1000,"sort":"authority"}' \
  > backlinks.json

# keep the 25 strongest ranked referring domains (requires jq)
jq '[
  .results[]
  | select(.cg_authority != null and .cg_authority >= 50)
] | sort_by(.cg_authority) | reverse | .[:25]' backlinks.json

cg_authority is a 0-to-100 percentile derived from Common Crawl harmonic centrality. it is useful for ordering a list; it is not a claim that a link will rank your page. some domains havenull authority because they are absent from the ranks file, which is why the filter checks for a value before comparing it. adjust the threshold to fit the size of your niche instead of treating 50 as a universal cutoff.

recipe 3: compare two competitors in Node

a free key cannot call the gap-analysis endpoint, but a small comparison is ordinary set math. fetch your own profile, fetch two competitors, remove every domain already linking to you, then rank the remainder by competitor overlap and authority. the script below uses three of the 15 monthly calls, so it can run five complete comparisons per month.

// save as compare-backlinks.js and run with Node 18+
const key = process.env.CG_KEY;
if (!key) throw new Error("set CG_KEY before running this script");

const mine = "yoursite.com";
const competitors = ["rival-a.com", "rival-b.com"];

async function getBacklinks(domain) {
  const response = await fetch("https://crawlgraph.com/api/v1/backlinks", {
    method: "POST",
    headers: {
      Authorization: "Bearer " + key,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ domain, limit: 1000, sort: "authority" }),
  });

  if (!response.ok) {
    throw new Error(domain + " returned HTTP " + response.status);
  }

  return (await response.json()).results;
}

async function main() {
  const myLinkers = new Set(
    (await getBacklinks(mine)).map((row) => row.linking_domain),
  );
  const candidates = new Map();

  for (const competitor of competitors) {
    for (const row of await getBacklinks(competitor)) {
      if (myLinkers.has(row.linking_domain)) continue;

      const candidate = candidates.get(row.linking_domain) ?? {
        linking_domain: row.linking_domain,
        cg_authority: row.cg_authority,
        competitors: [],
      };
      candidate.competitors.push(competitor);
      candidates.set(row.linking_domain, candidate);
    }
  }

  const ranked = [...candidates.values()].sort(
    (a, b) =>
      b.competitors.length - a.competitors.length ||
      (b.cg_authority ?? -1) - (a.cg_authority ?? -1),
  );

  process.stdout.write(JSON.stringify(ranked.slice(0, 25), null, 2) + "\n");
}

main().catch((error) => {
  process.stderr.write(String(error) + "\n");
  process.exitCode = 1;
});

this is a prospect shortlist, not an exhaustive link intersect. each request above compares the first 1,000 authority-sorted results. the endpoint accepts up to 10,000 rows if the larger response is useful, but the set math still only covers the slice returned by the API. the lifetime gap tool performs the deeper workflow and adds purpose-built filtering; free keys remain backlink-only.

recipe 4: feed an n8n or Zapier-style workflow

both automation tools can express the same three-step shape: accept a domain from a webhook, run an authenticated HTTP request, then pass the response rows to a sheet, database, or scoring step. this JSON is a provider-neutral blueprint—the placeholder syntax is intentionally generic, so map each field into the HTTP request and secret controls in the tool you use.

{
  "trigger": {
    "type": "webhook",
    "input": { "domain": "{{body.domain}}" }
  },
  "request": {
    "method": "POST",
    "url": "https://crawlgraph.com/api/v1/backlinks",
    "headers": {
      "Authorization": "Bearer {{secrets.CG_KEY}}",
      "Content-Type": "application/json"
    },
    "body": {
      "domain": "{{body.domain}}",
      "limit": 100,
      "sort": "authority"
    }
  },
  "output": {
    "items": "{{response.results}}"
  }
}

count executions before putting the workflow on a schedule. one webhook invocation is one backlink call. a weekly check of two domains costs 8 to 10 calls in a calendar month; an hourly trigger would exhaust the free tier on its first day. filter and store results downstream rather than repeating the same lookup for every transformation.

recipe 5: connect the hosted MCP server

the hosted endpoint at https://crawlgraph.com/mcp removes the local install step. Codex can read the bearer token from an environment variable and store only the endpoint plus variable name in its MCP config:

export CRAWLGRAPH_API_KEY="cg_live_<your-key>"
codex mcp add crawlgraph \
  --url https://crawlgraph.com/mcp \
  --bearer-token-env-var CRAWLGRAPH_API_KEY

clients use different shapes for remote MCP authentication. the MCP documentation has the separately tested Claude Code command; do not assume a Codex config can be pasted into another client unchanged. if the client only supports local stdio, use the open-source crawlgraph MCP server as the fallback and provide CRAWLGRAPH_API_KEY to that process:

npx -y crawlgraph-mcp

with a free key, ask the client to run a backlink lookup for one domain and return the strongest referring sites with the data fields intact. those tool calls draw from the same 15-call monthly quota as curl and Node. gap analysis and ranked outreach tools require the $99 lifetime tier, which raises the backlink allowance to 1,000 calls a month and adds 50 gap jobs.