moz's crawler explained: dotbot, rogerbot, and the crawled flag

moz runs two crawlers with two different jobs. dotbot is the web-wide crawler that builds moz's link index, the one domain authority is calculated from. rogerbot is the campaign crawler that powers site crawl inside a moz pro account. almost every confusing thing about moz crawling, a page that shows as not crawled, a DA that dropped, a site that never got picked up, comes from mixing the two up. here is each one, plainly.

what moz's crawler is called

There are two names, and knowing which one you are looking at solves most problems before you start troubleshooting. If you came here from a backlink question rather than a site-audit one, the wider guide on how to find backlinks for free covers the open-data route to the same link picture.

• dotbot is the link-index crawler. It crawls the open web, follows links between sites, and feeds the index behind Link Explorer, the Moz Links API, and the Domain Authority and Page Authority scores. It never knows or cares whether you have a Moz account.
• rogerbot is the campaign crawler. It only crawls sites that someone with a Moz account has pointed it at. It produces the site crawl issue report, not your DA.

Those two are the names Moz publishes, and they are the ones you will match on in a log file. You will also find third-party write-ups, log-parser bot lists, and older posts (including one on this site) that label the link crawler differently. Do not take any label on trust, this one included: read the user-agent string in your own access log and match on what is actually there.

Your server logs settle the question faster than any dashboard. Both crawlers name themselves in the user-agent string, so the first command below counts how many times each one hit you, and the second shows the most recent few with their timestamps, paths, and the full agent string.

# which moz crawler hit you, and how many times
grep -ioE 'rogerbot|dotbot' /var/log/nginx/access.log \
  | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn

# the most recent hits, with timestamp, path and full user-agent
grep -iE 'rogerbot|dotbot' /var/log/nginx/access.log | tail -5

Adjust the log path for your server. If neither name appears at all, that is itself the answer to “why is moz not crawling my site”, and the fix is further down.

what the “crawled” flag means

The crawled flag answers exactly one question: did moz's crawler fetch this specific URL and store what it found? It is not a quality judgment, not an indexing decision by Google, and not a ranking signal. A URL is either something the crawler has retrieved, with a timestamp attached, or it is a URL the crawler knows about but has not fetched.

That second state is the one people misread. A link can be discovered, because some other page pointed at it, and still sit uncrawled for weeks. Discovery and fetching are separate steps in every crawler ever built, moz's included.

Common reasons a page shows as not crawled:

• robots.txt blocks it. Check for a broad User-agent: * rule you forgot was there, not just a rule naming a crawler.
• no internal path. A campaign crawl starts at your seed URL and follows links. An orphan page with zero internal links in is unreachable, no matter how good it is.
• the only links to it are nofollowed or sit behind a form, a search box, or JavaScript that never renders a real <a href>.
• it returns a non-200 status, or sits at the end of a redirect chain the crawler gave up on.
• a canonical points somewhere else, so the crawler treats the other URL as the one worth keeping.
• the crawl stopped short. Compare the page count in your crawl summary against the number of URLs you expect. If the crawl fetched far fewer, the rest of the site was outside that run and depth or a limit is why.
• it is new. A page published yesterday has not had a crawl cycle yet. This is the single most common answer and the least satisfying one.

Work down that list in order. Six of the seven are things you can check in under a minute, and the seventh is solved by waiting.

what site crawl is on a moz pro account

Site Crawl is the technical-audit half of Moz Pro. You create a campaign for a domain, rogerbot walks the site from your seed URL outward, and Moz turns what it found into an issue report: broken pages, redirect chains, duplicate titles and content, missing meta descriptions, pages blocked by robots.txt or meta robots, and the like.

Two things about it are worth knowing up front. First, it is entirely separate from your link data: Site Crawl finding 400 issues does not change your DA, and a clean Site Crawl does not mean moz has crawled your backlinks. They are different crawlers writing to different places. Second, a campaign crawl reports on the slice of the site rogerbot actually reached, not the whole site, so if your important pages sit eight clicks deep they may never appear in it. The crawl summary in your account tells you how many pages the run fetched, which is the number to sanity-check against your sitemap.

This is the part of Moz that is genuinely good and often underrated. If you already pay for Moz Pro and only ever open Link Explorer, you are leaving the other half of the product on the shelf.

why moz has not crawled your url yet

Cadence is the sibling question, and it has its own post: how often each backlink index refreshes, and which one shows a new link first, is covered in how often backlink tools crawl. The short version for moz is that the two crawlers work on different clocks. A campaign crawl repeats on a schedule you can see in the campaign itself. The link index is not on a schedule you control at all: dotbot crawls continuously, moz republishes the index in batches, and DA and PA are recalculated when that happens.

The part worth spelling out here is the diagnosis behind “moz is not crawling my site, my PA and DA are stuck.” A link crawler discovers pages by following links. A brand new domain with nothing pointing at it gives dotbot no road in, so there is nothing to fetch and nothing to score. That is not moz being broken, it is an absence of evidence. One link from a site that is itself crawled, a directory, a profile, a partner page, opens the door. Submitting a URL to Google Search Console does nothing for moz; the two indexes share no plumbing.

how to stop or slow a moz crawl

First, confirm what is actually hitting you with the log commands above. People reach for this section over traffic that turns out to be a scraper wearing someone else's name.

If it is rogerbot, the crawl was started from a Moz account, so that account is the only place it can be ended at source. Look there for a stop, pause, or delete control on the campaign; what is available depends on your plan, so check rather than assume. Deleting or pausing the campaign is also what prevents the next scheduled run, which robots.txt alone will not do cleanly.

On your own side, robots.txt is the lever, with one caveat: it is not a mid-flight kill switch. Crawlers fetch robots.txt periodically and work from a cached copy between fetches, so a rule you ship now takes effect on their next check, not on the next request. Expect a lag, and do not assume a specific one.

The block below is the version that is safe to copy. It disallows nothing; it just states both crawlers explicitly, which is what you want when the real problem is a wildcard rule catching them by accident, or when you are about to change robots.txt and want moz's access pinned deliberately rather than inherited.

# safe to paste: every line here is either a comment,
# a user-agent name, or an allow. it disallows nothing.
# it makes moz's two crawlers explicit so a future wildcard
# rule cannot catch them by accident.
#
# note: a named group means these agents stop reading your
# "User-agent: *" group entirely. copy your own disallows into
# each group, in place of the commented example lines.

User-agent: dotbot
Allow: /
# Disallow: /cart/
# Disallow: /search

User-agent: rogerbot
Allow: /
# Disallow: /cart/
# Disallow: /search

The blocking rule is deliberately not behind a copy button on this page. If you do want to stop the campaign crawl, it is two lines, User-agent: rogerbot followed by Disallow: /, and you should type them yourself, because the identical two lines with dotbot in place of rogerbot do something completely different and the two get pasted interchangeably. Remove the rule once the crawl you were fighting is over. A forgotten Disallow: / for rogerbot is a quiet way to make your site crawl report go permanently empty.

Crawl-delay is worth a mention only to say it is not part of the robots.txt standard and not every crawler acts on it. Treat it as a request that may be ignored, and if you need a hard ceiling on request rate, rate-limit at your web server or CDN where you can actually observe the effect.

why your domain authority dropped or did not update

DA is a relative score, and that single word explains most of the confusion. It is scaled against the rest of moz's index, so your number can fall while your link profile is flat or even improving, simply because other sites in the comparison set gained more than you did. The scale is also compressed at the top: going from 20 to 30 is ordinary, going from 70 to 80 is not.

Reasons DA moves without you doing anything wrong:

• the index refreshed and the whole population was rescored
• moz changed how DA is calculated, which it has done more than once
• links that existed in the last index were not re-found in this one
• a big site that linked to you got dropped, redirected, or de-indexed
• a competitor gained referring domains faster than you did

Reasons DA does not update at all:

• the pages that link to you have not been crawled since the link went up
• those linking pages are blocked, noindexed, or unreachable
• you are between index refreshes, so nothing has been recalculated

Notice what every item on both lists has in common: it happens on other people's sites or inside moz's pipeline. Your own server is not where your authority score is decided.

The load-bearing point: google does not use domain authority. It is moz's model of ranking potential, useful as shared shorthand when you qualify a prospect or report to a client, and useless as a diagnosis of why your traffic changed last Tuesday. If DA moved and rankings did not, nothing is broken. The two are not wired together. The crawlgraph vs moz comparison goes deeper on where the DA metric earns its keep.

what a crawl-based index can and cannot tell you

Every backlink tool, moz included, is showing you one crawler's sample of the web. That is not a criticism, it is arithmetic. No crawler fetches every page, so two tools reporting different referring-domain counts for the same site are not arguing, they are describing different samples.

A crawl-based index reliably tells you: which pages the crawler fetched, when it fetched them, and which links it saw on the page at that moment. It cannot tell you what google saw, whether a link passes any value, or why a ranking moved. Treat the numbers as a floor and a direction of travel, never as a census.

The useful move is to check one closed index against an open one. The Common Crawl web graph is published openly by commoncrawl.org, so the crawl and the host-level link graph behind it can be inspected by anyone, rather than taken on trust. When two independent samples agree that a domain links to your competitor, that is a much stronger signal than either one alone.

crawlgraph runs on that open graph. The release it currently serves, Apr-Jun 2026, covers 121.1M domains and 3.90B domain-level links. Common Crawl publishes a new main crawl on a roughly monthly rhythm upstream; crawlgraph ingests and republishes a combined release on a roughly quarterly one, which is why the release ids are three-month spans. Figures from an older release will not match, so check which release a number came from before comparing two of them.

If you want to see which domains link to your competitors but not to you, the gap analysis runs without a subscription: an unpaid run asks for an email address, returns your first 5 gaps, and is rate limited per IP. The full list is on the paid plan, $99 once. There is also a free API key with 15 backlink lookups a month, no card, on the API docs page if you would rather script it. Neither replaces Site Crawl, which does a job we do not do at all.

faq

what is moz's crawler called?

Two crawlers, two names. dotbot crawls the open web and builds moz's link index, which is where Domain Authority comes from. rogerbot crawls one site at a time for a Moz Pro campaign and produces the site crawl issue report. Both name themselves in the user-agent string, so if a write-up you read uses some other label for the link crawler, go by what your access log actually contains.

what does the crawled flag mean in moz?

That moz's crawler fetched that exact URL and stored what it found, with a timestamp. Not crawled means the URL is known but has not been fetched, usually because robots.txt blocks it, no followed link reaches it, it returns a non-200 status, the crawl stopped before reaching it, or it is simply too new.

how often does moz crawl my url?

A campaign site crawl repeats on the schedule shown in the campaign. The link index is different: dotbot crawls continuously and moz republishes the index in batches, so a heavily-linked page is revisited often and a page with no inbound links may not be fetched at all. The full cadence comparison across moz, ahrefs, semrush, majestic and Common Crawl is in the separate post on how often backlink tools crawl.

can i cancel a moz crawl test that is already running?

“Crawl test” is the phrase people use for a moz-run crawl of their own site. Start by confirming from your access log that it really is rogerbot and not a scraper borrowing the name. If it is rogerbot, it was started from a Moz account, so that account is where to look for a stop, pause, or delete control on the campaign, and what you get there depends on the plan. On your side, a Disallow rule for rogerbot in robots.txt takes effect on the crawler's next robots.txt check rather than immediately, and a short run is often cheaper to let finish than to fight. Moz support can end a run at their end if you genuinely cannot wait it out.

does blocking rogerbot hurt my seo?

No. rogerbot only feeds your own Moz Pro reports, so blocking it costs you the site crawl and nothing else. Blocking dotbot costs more, but not what people think: it stops moz fetching your pages, so your URLs and the links you point outward fade out of the link index. It does not remove the inbound links other sites give you, because those are found by crawling those sites, not yours. Neither rule affects Google.

why did my da drop when i did not lose any links?

Because DA is scored relative to every other site in moz's index. When the index refreshes, everyone is rescored together, so a flat link profile in a rising field shows up as a lower number. Changes to the DA model itself do the same thing. Google does not use DA, so a drop there is not a ranking problem.

moz is not crawling my site at all. what now?

Check robots.txt for a rule that catches dotbot or the wildcard user-agent, confirm your pages return 200, and confirm something crawlable links to you. A brand new domain with zero inbound links has nothing for a link crawler to follow, and no amount of waiting fixes that. One link from a site that is itself crawled is what opens the door.