about a quarter of robots.txt files at top sites block the big ai training crawlers. we read robots.txt for the 1,000 highest-ranked domains in the tranco list on 30 october 2026, and 493 returned a readable file. of those, 26.8% block ccbot, 25.2% block claudebot and 24.9% block gptbot, against 5.9% for googlebot. sites treat training and search differently: 46 block gptbot but not oai-searchbot, and none do the reverse.
we run on common crawl data, so ccbot matters to us directly, and we explain the consequences for backlink data below. crawlgraph's own robots.txt does not block any of these crawlers.
which ai crawlers are blocked most
a site counts as blocking a crawler when its rules disallow an ordinary page for that crawler, either in a group naming it or through a User-agent: * rule that applies because it isn't named.
| crawler | operator | blocked sites | share of 493 | blocked by name |
|---|---|---|---|---|
| Bytespider | bytedance | 136 | 27.6% | 89 |
| CCBot | common crawl | 132 | 26.8% | 85 |
| ClaudeBot | anthropic | 124 | 25.2% | 83 |
| GPTBot | openai | 123 | 24.9% | 84 |
| meta-externalagent | meta | 115 | 23.3% | 68 |
| Google-Extended | 109 | 22.1% | 71 | |
| Applebot-Extended | apple | 108 | 21.9% | 65 |
| PerplexityBot | perplexity | 104 | 21.1% | 66 |
| Amazonbot | amazon | 101 | 20.5% | 56 |
| anthropic-ai | anthropic | 101 | 20.5% | 50 |
| Claude-User | anthropic | 89 | 18.1% | 45 |
| ChatGPT-User | openai | 88 | 17.8% | 47 |
| Claude-SearchBot | anthropic | 85 | 17.2% | 40 |
| OAI-SearchBot | openai | 77 | 15.6% | 36 |
| Bingbot | microsoft | 31 | 6.3% | 3 |
| Googlebot | 29 | 5.9% | 5 |
- no block
- partly restricted
- blocked by a catch-all rule
- blocked by name
CCBot common crawl
GPTBot openai, training
ClaudeBot anthropic, training
Google-Extended google, ai control token
OAI-SearchBot openai, search
Googlebot google, search
the gaps between these rates are small, so don't read too much into the order among the top few. the wider point is that the big training crawlers are blocked about four times as often as googlebot or bingbot, and many of the blocks on those two come from infrastructure domains that disallow nearly everything.
ranks made little difference below the top 100: ccbot was blocked on 25.7% of readable files ranked 101 to 500, and 25.6% ranked 501 to 1,000. the top 100 reached 36.5%, but that is 19 of only 52 readable files, and most of those blocks came from a catch-all rule rather than a named one.
sites separate training from search
openai and anthropic both list separate crawlers for training and for search, and site owners are using that split:
- openai. 46 sites block gptbot but not oai-searchbot, 77 block both, and none block only oai-searchbot.
- anthropic. 39 sites block claudebot but not claude-searchbot, 85 block both, and none block only claude-searchbot.
- google. 80 sites block google-extended but not googlebot. no site blocks googlebot while leaving google-extended open.
a block isn't always a statement about ai. 75 sites block all five of ccbot, gptbot, claudebot, google-extended and perplexitybot, and many do it through a catch-all rule. 148 block at least one of those five.
who blocks ccbot by name, and who doesn't
on 30 october these sites had a group naming ccbot with a full disallow: linkedin.com, amazon.com, tiktok.com, yahoo.com, nytimes.com, cnn.com, theguardian.com, bbc.com, ebay.com, who.int, nature.com, bloomberg.com, washingtonpost.com, wired.com and tripadvisor.com. chatgpt.com also names ccbot, while claude.ai names gptbot.
wikipedia.org, microsoft.com, apple.com, openai.com and anthropic.com did not restrict ccbot, and github.com's rules for gptbot and claudebot were partial. robots.txt files change often, so check the current file before relying on any of these.
what each crawler is for
these descriptions come from each operator's own documentation, read on 30 october. blocking one crawler doesn't necessarily affect the others from the same company.
| crawler | purpose, per the operator | effect of blocking |
|---|---|---|
| CCBot | builds common crawl's open web archive | crawling stops; the faq doesn't say earlier data is removed |
| GPTBot | content that may train openai's models | signals the content should not be used for training |
| OAI-SearchBot | surfacing sites in chatgpt search | openai recommends allowing it to appear there |
| ChatGPT-User | actions a chatgpt user starts | openai says robots.txt rules may not apply |
| ClaudeBot | content that could contribute to anthropic's training | signals future material should be excluded from training |
| Claude-SearchBot | improving search results for users | may reduce visibility in user search results |
| Claude-User | fetching pages when a user asks | may reduce visibility in user-directed search |
| Google-Extended | a control token for gemini training and grounding | google says no effect on search inclusion or ranking |
how to block or allow them
decide on two questions separately: should your content train models, and do you want to appear in ai search? this combination blocks training and stays eligible for search. 20 of the 493 readable files do exactly this.
# block model training, stay eligible for ai search User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: /
check three things before you ship it. a crawler follows only its most specific group, so a named group replaces your * rules for that bot. robots.txt applies per host, so each subdomain needs its own file. anthropic's page makes the same point. and listing a crawler doesn't mean it will be seen in your logs, because common crawl warns that other crawlers sometimes impersonate ccbot, so verify requests before you act on them.
ai visibility is a separate question from access. our guide to answer engine optimization covers what makes a page citable once a crawler can reach it, and ai search referral tracking covers how to measure the visits.
what blocking ccbot means for backlink data
common crawl's web graphs are built from its crawls, and crawlgraph is built on those graphs. so if ccbot can't read your pages, we would expect the links on those pages to be missing or to fade from later graphs. links pointing at your site are a different matter, because they come from other sites' pages.
that matches what we see for incoming links. nytimes.com blocks ccbot, and it still has 433,432 referring domains in the apr-jun 2026 graph that crawlgraph serves. bbc.com, which also blocks it, has 257,756.
to see which sites link to a domain in the current graph, try the free crawlgraph check. for more on the data itself, read what common crawl is.
how we measured this
we took the top 1,000 domains from tranco list PYK6J and requested https://domain/robots.txt on 30 october 2026, trying the www host if the first attempt failed. we identified ourselves in the user agent and made one request per host. we then applied rfc 9309 group matching and longest-match rules to the test path /a-typical-page/ for each crawler.
- 493 files were readable and form the base. 73 domains returned 404, meaning no file and so no restriction. the other 434 could not be read: 290 did not respond, which is common for infrastructure domains in this list, 76 refused our request, and the rest returned an error or something that was not a robots file.
- tranco ranks domains, not just websites. the list includes cdns and dns providers, some of which disallow nearly everything for all crawlers.
- a stated rule is not proof of behaviour. we did not test whether any crawler obeys these rules.
- we can only see robots.txt. sites may also block at the server or cdn, serve different files to different visitors, or use different rules on subdomains.
- it is one snapshot. we made one read on one day and did not measure change over time.
the full results are available as a csv. use them however you like; a link back to this page is appreciated.
frequently asked questions
faq
what is ccbot?
ccbot is the web crawler run by common crawl, a non-profit that publishes an open archive of web crawl data. it identifies itself as CCBot/2.0, checks robots.txt before fetching a page and obeys crawl-delay. common crawl also publishes its ip ranges so you can verify real requests, because other crawlers sometimes pretend to be ccbot.
how do i block ccbot?
add a group to your robots.txt with user-agent: ccbot and disallow: /. common crawl says its crawler will stop crawling your site and will keep checking the file for changes. its faq does not say that content already collected is removed. it also runs a separate opt-out registry.
does blocking gptbot remove my site from chatgpt search?
not according to openai. its documentation says gptbot concerns model training, oai-searchbot concerns appearing in chatgpt search, and the two settings are independent. to block training while remaining eligible for search, disallow gptbot and allow oai-searchbot.
what is claudebot?
claudebot is anthropic's crawler for collecting web content that could contribute to model training. anthropic lists two others: claude-searchbot, which indexes content for search, and claude-user, which fetches pages when a user asks. anthropic says blocking those two may reduce your visibility in user search results.
will blocking google-extended hurt my google rankings?
google says it will not. its documentation says google-extended does not affect a site's inclusion in google search and is not used as a ranking signal. it is a control token for gemini training and grounding, so it needs no separate crawler of its own.
does robots.txt actually stop ai crawlers?
it is a request, not a technical barrier. the operators named here say they honour it for the listed crawlers, with some exceptions: openai says robots.txt rules may not apply to user-initiated chatgpt-user requests. to enforce a block you need server or network rules, and blocking by ip has its own failure modes.
documents crawlgraph backlink workflows, open-data methods, and the limits of each report.
plus one when a new common crawl release lands. that is all.