What is AhrefsBot: The Complete Guide to Ahrefs' Web Crawler

If you've spotted AhrefsBot in your server logs, your Cloudflare firewall events, or a robots.txt audit, you're looking at one of the most active web crawlers on the internet. It's not malware. It's not a scraper. It's not trying to harm your site.

AhrefsBot is the proprietary crawler that powers Ahrefs' SEO data set - the backlink index, keyword data, organic traffic estimates, and competitive intel that hundreds of thousands of SEO teams use daily.

But "it's legitimate" doesn't settle the policy decision. You still need to decide whether to allow it, restrict it, or block it outright, and each option carries tradeoffs most site owners miss. Blocking AhrefsBot doesn't just affect your own Ahrefs dashboard. It affects every agency, every competitor analyst, and every link builder who uses Ahrefs to track links pointing to your site. For agencies running backlink campaigns, that distinction is operational. And most guides skip it.

That policy decision is what this guide addresses. We cover what AhrefsBot is, how it differs from AhrefsSiteAudit, how to verify it's real (and not a spoofed bot), how to control it via robots.txt, CDN rules, and IP blocking, and what you give up when you block it. This is written for SEO managers, technical marketers, and agency owners who need more than a surface-level overview.

What Is AhrefsBot? The Web Crawler Behind Ahrefs' SEO Data

AhrefsBot is a web crawler operated by Ahrefs Pte. Ltd., the Singapore-based company behind one of the most widely used SEO toolsets in the industry. Its job is simple: crawl publicly accessible pages, follow links, and send what it finds back into Ahrefs' search index. That index powers Site Explorer's backlink database, the organic keywords report, and Domain Rating.

According to Ahrefs' own SEO glossary, AhrefsBot crawls about 5 million pages per minute, putting it among the largest non-search-engine crawlers on the web. In day-to-day log reviews, it's one of the few bots you'll see at real scale. The crawl volume is the reason Ahrefs can keep a live backlink index instead of relying on an old snapshot.

The bot identifies itself with a specific user-agent string. The current format is:

Mozilla/5.0 (compatible; AhrefsBot/7.0; +https://ahrefs.com/robot)

That +https://ahrefs.com/robot value is intentional. It sends server admins to Ahrefs' official robot documentation page, where they publish IP ranges, verification steps, and opt-out guidance. That's standard behavior for reputable crawlers - Googlebot follows the same pattern, with a reference URL embedded in the user-agent so anyone can validate what they're seeing in logs.

AhrefsBot also powers Yep, Ahrefs' search engine that launched publicly in 2022. That shared infrastructure matters for access control. Allow AhrefsBot, and your pages can show up in Ahrefs' index and Yep's search results.

The crawler is Cloudflare Verified, which means Cloudflare's bot management classifies AhrefsBot as a verified bot rather than suspicious traffic. Cloudflare reserves that tier for bots with consistent behavior and an identity that can be checked, so this isn't a vanity label. If you're using Cloudflare Bot Management, you'll see AhrefsBot grouped with major crawlers like Googlebot and Bingbot.

AhrefsBot is also an IndexNow partner. IndexNow lets sites notify crawlers about new or updated URLs so they don't have to wait for the next scheduled crawl. If you're publishing or refreshing pages and you care about how fast those changes appear in Ahrefs, IndexNow support helps shorten that lag.

AhrefsBot vs AhrefsSiteAudit: Two Crawlers, Two Very Different Jobs

Most guides blur this line, and that causes real problems when you're setting bot access rules.

AhrefsBot and AhrefsSiteAudit are separate crawlers with their own user-agent strings, their own IP ranges, and different jobs. Treat them as the same bot - or block one when you meant the other - and you get the wrong result fast.

AhrefsBot is the public web crawler. It crawls the open web on its own. Nothing in your Ahrefs account triggers it. It discovers your site through links from other pages, follows those links, and indexes what it finds. That data goes into Ahrefs' shared index - the one every Ahrefs subscriber uses in Site Explorer for any domain.

AhrefsSiteAudit works differently.

It's a user-triggered crawler that runs only when an Ahrefs subscriber starts a Site Audit for a specific domain. If you're auditing your own site in Ahrefs' Site Audit tool, AhrefsSiteAudit is the crawler doing the crawl. Its user-agent string is distinct:

Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +https://ahrefs.com/robot)

Those differences change what you should allow or block.

A site owner who wants to audit their own site in Ahrefs needs to allow AhrefsSiteAudit in robots.txt and any WAF rules. At the same time, that same team may want to restrict AhrefsBot's autonomous crawling for bandwidth or competitive reasons. They're separate decisions - but only if you recognize there are two crawlers.

One more difference matters in server logs: crawl-delay behavior during JavaScript rendering. Ahrefs says crawl-delay directives can't be enforced during JavaScript rendering passes because multiple assets are fetched at the same time to mirror real browser behavior. That shows up with AhrefsBot on JS-heavy pages. If crawl-delay is your main control for server load and your pages lean hard on JavaScript, expect more concurrent requests than the crawl-delay value implies. We'll cover that in more detail in the crawl behavior section.

Why AhrefsBot Is Crawling Your Site (And What It's Actually Collecting)

AhrefsBot is on your site because another site linked to it. That's the short answer.

Those links are the whole entry point. The crawler follows hyperlinks across the web, similar to how Google's crawl systems work. Once Ahrefs has a page in its index that links to your domain, AhrefsBot follows that link, hits your site, and crawls onward. No submission. No verification. No account action. If your site is public and has inbound links from pages Ahrefs crawls, AhrefsBot finds it.

What it collects is more specific. The main data points AhrefsBot gathers include:

• Backlink data - every outbound link on a crawled page, including anchor text, link type (follow/nofollow/sponsored), and surrounding content context
• Page-level signals - HTTP status codes, canonical tags, meta robots directives, page titles, and structured data
• Internal link structure - how pages within a domain connect to each other, which informs Ahrefs' internal link reports
• Content signals - on-page content used to determine topical relevance and keyword associations
• Technical indicators - page speed signals, redirect chains, and response headers

That crawl data feeds the metrics Ahrefs users care about: Domain Rating (DR), URL Rating (UR), backlink counts, referring domain counts, and organic keyword estimates. If we're running competitor research in Ahrefs, we're looking at outputs built from AhrefsBot crawls of those competitor pages.

For site owners, the takeaway stays simple. When someone earns a backlink and checks it in Ahrefs, that check only works if AhrefsBot has crawled the linking page recently. A link that exists but hasn't been crawled yet won't show up. A page that blocks AhrefsBot outright never contributes its outbound links to Ahrefs' dataset - so links from that page stay invisible inside Ahrefs. Understanding key link building metrics like DR and UR helps explain why crawl access matters so much to how your backlinks are valued.

Crawl frequency drives how quickly those links appear. Pages with high URL Rating and lots of inbound links get crawled more often. A freshly published guest post on a mid-authority site often takes days or weeks to show up in Ahrefs' backlink index, while a link from a high-DR news site can appear within hours.

How to Verify AhrefsBot Traffic Is Legitimate (Not a Fake Bot)

User-agent strings are easy to spoof. A scraper can send requests with AhrefsBot/7.0 in the header and your server will log it like the real crawler unless we validate the request. This isn't theoretical - bot impersonation is a common, documented tactic.

The right approach is reverse DNS lookup, using the same two-step process Google documents for Googlebot in their crawler verification guide.

Here's the workflow we use:

1. Pull the IP address from the server log entry you want to verify
2. Run a reverse DNS lookup on that IP address (host [IP] on Linux/Mac, or use an online tool)
3. Confirm the returned hostname ends in ahrefs.com or ahrefs.net
4. Run a forward DNS lookup on that hostname and confirm it resolves back to the original IP address
5. Cross-reference the IP against Ahrefs' published IP ranges

Both checks have to pass.

A hostname that ends in ahrefs.com but doesn't map back to Ahrefs' published IP ranges is a red flag. An IP inside Ahrefs' published range that doesn't reverse-resolve to an ahrefs.com hostname is also a red flag. Real AhrefsBot traffic clears both checks consistently.

This two-step reverse DNS verification matters because:

• User-agents are trivially spoofed - any HTTP client can send any user-agent string
• IP addresses are harder to fake - but attackers still proxy or rotate them at scale
• DNS records are authoritative - only Ahrefs controls what their IP ranges resolve to

That DNS point is the same one Google makes. Google's position is clear: reverse DNS verification is the only reliable way to validate Googlebot, and user-agent matching doesn't count. AhrefsBot works the same way. If we're making security decisions - especially inside a WAF or bot management tool - user-agent rules alone don't hold up.

For most teams that just want to confirm log activity is legitimate, a spot-check works fine. Run the reverse DNS + forward DNS loop on a few IPs. If each one resolves to an ahrefs.com or ahrefs.net hostname and maps back to the same IP, it's real AhrefsBot traffic.

Should You Block AhrefsBot? The SEO Trade-Off Most Site Owners Miss

Most guides shrug and say "it's your choice." That answer dodges the operational impact. Blocking AhrefsBot has real consequences, and site owners usually notice them after the block is already in place.

The case for blocking is valid in a few situations. Private web apps, members-only portals, and sites with proprietary content don't gain much from third-party crawling. If the infrastructure is tight - think a small VPS serving a high-traffic site without a CDN - cutting non-essential crawl load is a sensible capacity move. Competitive intelligence also comes up: some businesses don't want competitors pulling their link profile or content signals from Ahrefs.

The case against blocking is stronger than most people expect, and it shows up in two places.

First, your own Ahrefs data takes the hit. Once AhrefsBot can't crawl the site, your pages stop updating in Ahrefs' index. Domain Rating, URL Rating, and organic keyword data drift as the index goes stale. Over time, Ahrefs stops being a reliable place to audit backlinks, track rankings, or monitor SEO health for that site.

That stale index leads straight into the second issue: third-party reporting. This is where blocks create the biggest mess, especially for teams working with an agency.

If a client site blocks AhrefsBot, it creates a reporting blind spot. When Rhino Rank builds backlinks for a client, we validate placements in Ahrefs: the linking page has been crawled, the link appears in the index, and anchor text plus link attributes are recorded correctly. But if the client's site blocks AhrefsBot, links pointing to that site may never show up in Ahrefs' backlink index - even when the linking page itself is crawled. Ahrefs needs to crawl both the linking page and the destination URL to process the relationship cleanly. This is one reason our managed service includes ongoing placement verification rather than a one-time check.

Once that happens, everything downstream degrades. ROI reporting becomes unreliable. Link velocity tracking breaks. Competitive benchmarking disappears.

The bottom line: for any public-facing site that uses Ahrefs or works with an agency that does, blocking AhrefsBot is almost always the wrong decision. Rate-limit it. Control crawl frequency. But don't block it outright unless there's a specific, defensible reason.

How to Block AhrefsBot in robots.txt: Exact Directives and When to Use Each

If we've chosen to restrict AhrefsBot, robots.txt is the standard starting point. AhrefsBot follows robots.txt directives. Ahrefs documents that behavior in its crawler policies. Below are the exact directives for common setups.

To block AhrefsBot from your entire site:

User-agent: AhrefsBot
Disallow: /

To block AhrefsSiteAudit from your entire site (separate directive):

User-agent: AhrefsSiteAudit
Disallow: /

To block both Ahrefs crawlers from your entire site:

User-agent: AhrefsBot
Disallow: /

User-agent: AhrefsSiteAudit
Disallow: /

To block AhrefsBot from a specific directory only:

User-agent: AhrefsBot
Disallow: /members/
Disallow: /private/
Disallow: /admin/

To add a crawl-delay for AhrefsBot (rate limiting without blocking):

User-agent: AhrefsBot
Crawl-delay: 10

This instructs AhrefsBot to wait 10 seconds between requests. Adjust the value to match server capacity. In practice, 5-10 seconds usually cuts load without tanking crawl coverage.

Critical caveat on crawl-delay: As covered in the AhrefsBot vs AhrefsSiteAudit section, Ahrefs documentation states crawl-delay won't apply during JavaScript rendering passes. When AhrefsBot renders a JavaScript-heavy page, it pulls multiple assets at once to mimic real browser behavior, so request concurrency can spike beyond what crawl-delay implies. If our site relies heavily on JavaScript and we're using crawl-delay to manage load, plan for that gap - add IP-level rate limiting alongside robots.txt, or serve AhrefsBot a pre-rendered HTML version where possible.

To allow AhrefsBot everywhere except one section:

User-agent: AhrefsBot
Disallow: /confidential-section/
Allow: /

The Allow directive overrides Disallow on more specific paths. Use this pattern to block one folder without locking the bot out of the whole site.

One important note: robots.txt is a voluntary protocol. Legit crawlers like AhrefsBot respect it. Scrapers won't. If we're dealing with an impersonator spoofing the AhrefsBot user-agent, robots.txt won't stop that traffic. At that point, we need IP-level controls, which we cover next.

For the full list of user-agent strings and official robots.txt guidance, refer to Ahrefs' robot documentation directly.

Blocking AhrefsBot by IP Address: When robots.txt Isn't Enough

IP-level blocking fits when robots.txt can't do the job - either because the traffic spoofs the user-agent, or because we need enforcement at the infrastructure layer instead of relying on crawler compliance.

Ahrefs publishes its IP ranges at https://www.ahrefs.com/robot. Treat that page as the source of truth. Skip third-party IP lists. They age fast as Ahrefs expands and shifts infrastructure, so they create blind spots. Pull the ranges straight from Ahrefs.

To block AhrefsBot at the server level using Apache, add the following to your .htaccess file or server configuration:

<RequireAll>
  Require all granted
  Require not ip [AHREFS_IP_RANGE_1]
  Require not ip [AHREFS_IP_RANGE_2]
</RequireAll>

Replace the placeholder values with the current IP ranges from Ahrefs' documentation.

For Nginx, use a deny approach inside the relevant server block:

geo $block_ahrefs {
  default 0;
  [AHREFS_IP_RANGE] 1;
}

server {
  if ($block_ahrefs) {
    return 403;
  }
}

There's a real difference between returning a 403 and returning a 429. Ahrefs notes that non-200 status codes signal the crawler to adjust crawl rate. A 429 (Too Many Requests) tells AhrefsBot to slow down. A 403 (Forbidden) tells it to stop. If the goal is throttling rather than a full block, 429 responses from those IP ranges give us finer control than a hard 403.

IP blocking needs upkeep. Ahrefs' ranges change as it scales. If we block at the IP level, we also need a process to re-check Ahrefs' published IP list and refresh our rules on a schedule - otherwise new Ahrefs IPs slip through.

Blocking AhrefsBot via CDN or WAF (Cloudflare and Others)

If your site sits behind Cloudflare, you've got two clean ways to control AhrefsBot traffic: Cloudflare's built-in bot controls, or your own firewall rules.

Option 1: Cloudflare Bot Management (Enterprise/Pro)

AhrefsBot is on Cloudflare's verified bots list. In Bot Management, you can allow or block verified bots as a group, or pick off AhrefsBot specifically. On Cloudflare Pro and above, a firewall rule can match cf.bot_management.verified_bot and then apply the action you want to AhrefsBot traffic.

Option 2: Custom Firewall Rules

Need tighter control. Build a Cloudflare firewall rule around this logic:

• Field: User Agent
• Operator: contains
• Value: AhrefsBot
• Action: Block or Challenge or JS Challenge

User-agent matching alone is easy to spoof, so don't stop there if you care about accuracy. Add IP range matching via Cloudflare's IP Source field. That combo lowers two common failure modes: blocking real users because of sloppy matching, and missing spoofed bots that rotate user-agents while staying in known crawler ranges.

Other WAF and CDN platforms (Fastly, Sucuri, AWS WAF, Imperva) support the same pattern. Match the user-agent and/or IP range, then choose an action. Test in "log only" or "challenge" first, then move to hard blocks once you've confirmed you're not catching anything you need.

Why AhrefsBot Isn't Crawling Your Site (And How to Fix It)

The previous sections cover how to restrict AhrefsBot. Plenty of website owners run into the reverse issue: they want more crawl activity, and AhrefsBot barely shows up.

That gap hurts most right after you publish new pages or land fresh backlinks and need them reflected in Ahrefs fast. New domains feel it too. Thin link profiles mean low authority signals, and AhrefsBot allocates crawl budget accordingly, so low-DR sites get fewer revisits.

The most common reasons AhrefsBot isn't crawling your site:

1. Your robots.txt is blocking it (possibly unintentionally)

Start with robots.txt. Look for Disallow: / under a wildcard user-agent like User-agent: *. If you've blocked all bots, AhrefsBot gets blocked too. This misconfiguration happens a lot - teams drop in a catch-all to slow down scrapers, then accidentally shut out legitimate bots.

2. Your site has very few inbound links

AhrefsBot finds pages through links. A new domain with no backlinks is invisible to the crawler. The fix isn't technical - you need links from pages AhrefsBot already crawls often. Once a high-authority page that gets crawled regularly links to you, AhrefsBot follows that path and discovers the domain. If you're starting from scratch, our guide to link building for new websites covers a practical 90-day approach to building that foundation.

3. Your server is returning error codes

Crawler access dies fast when the server falls over. If your server returns 5xx responses or times out during crawl attempts, AhrefsBot will push your site down the queue. Pull server logs and look for patterns like 503, 504, or 500 during bot traffic. Fix those issues, and crawl success rates go up - which usually brings more frequent revisits.

4. You're blocking Ahrefs' IP ranges in your WAF or CDN

WAF rules often cause this one. Some security setups - especially aggressive managed rulesets - block large data center ranges by default. Ahrefs crawlers run from data center IPs, not residential IPs, so they trip those filters. Check Cloudflare firewall events or your WAF logs for blocked requests coming from Ahrefs' IP ranges.

5. Using IndexNow to accelerate crawl discovery

If you want the fastest legitimate path into Ahrefs' index, implement the IndexNow protocol. AhrefsBot is an IndexNow partner, so an IndexNow submission pings AhrefsBot (and other participating crawlers) that a URL is new or updated. Most major CMS platforms already have IndexNow plugins. This doesn't guarantee an instant crawl, but it cuts discovery lag compared to waiting on link discovery alone.

AhrefsBot and SEO Data Quality: Why Crawl Access Affects Your Ahrefs Reports

Most guides skip this part. Agencies feel it fast.

Every metric you see in Ahrefs - Domain Rating, URL Rating, backlink count, referring domains, organic traffic estimates - comes from pages AhrefsBot managed to crawl. Block that access and the dataset starts drifting. It rarely breaks in a clean, obvious way on day one. It just gets less reliable, then keeps sliding as time passes.

That reliability problem shows up first in your own reporting.

How blocking affects your own site's metrics:

If AhrefsBot can't crawl your site, Ahrefs can't refresh what it knows about your URLs. New pages won't get picked up. Links you earn won't get validated against a live, crawlable destination. And because Domain Rating depends on Ahrefs confirming real link relationships to accessible pages, a block breaks that feedback loop. Ahrefs can't confirm what it can't fetch.

For a mid-market SaaS company spending $3,000 per month on link building, this turns into a budget problem, not a tooling quirk. If the client blocks AhrefsBot, the agency can't verify placements in Ahrefs, can't prove link velocity, and can't show the link-to-rank movement in the same reporting view. The work can be solid. The reporting stays blind.

Blind reporting also hits competitor work.

How blocking affects competitor analysis:

If a competitor blocks AhrefsBot, their site turns into a black box inside Ahrefs. Backlink data thins out. Content discovery suffers. Keyword visibility gets patchy. That cuts both ways: blocking AhrefsBot creates competitive opacity, and it also removes a clean benchmark in Ahrefs for that competitor.

That benchmarking gap turns into a delivery issue for agencies.

The link building agency reporting problem:

When Rhino Rank delivers a link placement report, we verify each placement in Ahrefs. We confirm the linking page has been crawled, the link is indexed, the anchor text matches the brief, and the DR of the linking domain sits within the agreed range. If the client's site blocks AhrefsBot, we can still verify the linking page - but we can't confirm that Ahrefs processed the relationship between the linking page and the client's domain. Links may show up in the linking page's outbound link report but not in the client's inbound link report.

That split is the problem. The link is live. The placement is real. But the tool both sides rely on can't see the connection, so the reporting doesn't line up. This is one reason our managed service clients are advised to keep AhrefsBot access open - it keeps reporting clean on both sides.

The fix is simple: allow AhrefsBot to crawl your site. If you have legitimate reasons to restrict it, use crawl-delay instead of a full block. Partial access beats zero access for data quality.

AhrefsBot's Crawl Behavior: How It Manages Server Load

Server load is usually the reason teams block bots in the first place. AhrefsBot is built to keep that load under control, and Ahrefs' published policies spell out the mechanics.

Crawl-delay compliance: AhrefsBot reads and respects Crawl-delay directives in robots.txt. If you set a crawl-delay, the bot will honor it between sequential requests. One exception matters here, and most competitor guides skip it: JavaScript rendering. When AhrefsBot renders a JS-heavy page, it pulls multiple assets at the same time so the page loads like it would in a real browser. During those rendering passes, crawl-delay doesn't apply because asset fetching runs in parallel. Sites running heavy JavaScript frameworks (React, Angular, Vue single-page applications) will see bursty request patterns in logs during rendering, even with crawl-delay set.

Those bursts are easier to control with response codes than with a single static setting.

Response code signals: AhrefsBot treats HTTP response codes as crawl-speed signals. A 429 (Too Many Requests) response tells the crawler to back off and reduce its request rate. A 503 (Service Unavailable) with a Retry-After header tells it to pause and retry later. Use these when you want to throttle based on real load, not a fixed crawl-delay that stays the same on quiet and busy days. For a full breakdown of what each status code means and how crawlers interpret them, see our complete HTTP status codes guide.

Load also drops once the bot has your assets.

Asset caching: AhrefsBot caches static assets (images, CSS, JavaScript files) to cut repeat requests on return visits. That means later crawl passes use less bandwidth than the first pass, since the bot skips unchanged static resources.

What gets revisited - and how often - depends on authority.

Crawl prioritization: Ahrefs prioritizes crawl frequency based on a page's link authority. High-DR pages with lots of inbound links get crawled more often. Fresh content on low-authority sites can sit for days or weeks before a revisit, while a page on a major news site may get recrawled within hours of an update. Search Engine Journal's overview of how web crawlers work covers the broader mechanics of crawl prioritization across different search and SEO crawlers if you want more context on how these systems allocate crawl budget.

Frequently Asked Questions

What is AhrefsBot and what does it do?

AhrefsBot is a web crawler operated by Ahrefs Pte. Ltd. It crawls public web pages on its own, follows links, and collects data for Ahrefs' SEO tools. That data feeds the backlink index, Domain Rating calculations, keyword datasets, and organic traffic estimates. It also powers Yep, Ahrefs' own search engine.

Ahrefs says AhrefsBot crawls about 5 million pages per minute, which puts it among the largest non-search-engine crawlers on the web.

What is the AhrefsBot user-agent string?

The current AhrefsBot user-agent string is Mozilla/5.0 (compatible; AhrefsBot/7.0; +https://ahrefs.com/robot).

That embedded URL points to Ahrefs' robot documentation page. It lists IP ranges, verification steps, and control options.

AhrefsSiteAudit is a separate, user-triggered crawler and it uses a different string: Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +https://ahrefs.com/robot).

How do I block AhrefsBot using robots.txt?

Add the following to your robots.txt file to block AhrefsBot across the entire site:

User-agent: AhrefsBot
Disallow: /

To block AhrefsSiteAudit separately, add a second directive:

User-agent: AhrefsSiteAudit
Disallow: /

If you want to slow AhrefsBot down instead of blocking it, use crawl-delay:

User-agent: AhrefsBot
Crawl-delay: 10

Crawl-delay won't be enforced during JavaScript rendering passes. JS-heavy sites can still see bursty request patterns during those renders.

What is the difference between AhrefsBot and AhrefsSiteAudit?

AhrefsBot is an autonomous public web crawler. It crawls the open web without any user action and builds Ahrefs' shared backlink and keyword index.

AhrefsSiteAudit is user-triggered. It runs only when an Ahrefs subscriber starts a Site Audit for a specific domain.

They use different user-agent strings, run from different IP ranges, and they exist for different jobs. If you run Ahrefs' Site Audit tool on your own site, you need to allow AhrefsSiteAudit - blocking it will cause your audits to fail.

How can I verify that a bot is really AhrefsBot and not a fake?

User-agent strings are easy to spoof, so user-agent matching alone isn't enough. Use reverse DNS lookup to confirm legitimacy - the same approach Google recommends for validating Googlebot.

Start with the IP address in your server logs. Run a reverse DNS lookup and confirm the hostname ends in ahrefs.com or ahrefs.net. Then run a forward DNS lookup on that hostname and make sure it resolves back to the original IP. Last step: cross-check that IP against Ahrefs' published ranges at https://www.ahrefs.com/robot.

Legitimate AhrefsBot traffic passes all three checks. Impersonators don't.

What happens to my Ahrefs data if I block AhrefsBot?

Blocking AhrefsBot weakens your Ahrefs data over time. New pages you publish won't get picked up into Ahrefs' index. Backlinks pointing to your site may never get processed into the backlink database. Domain Rating and URL Rating drift because they're working from stale crawl data.

Stale crawl data also creates reporting problems. If your link building agency uses Ahrefs for verification - like Rhino Rank - a full block means earned backlinks can't be confirmed in Ahrefs, which breaks reporting.

For most public-facing sites, crawl-delay is the better call than a full block.

How do I block AhrefsBot in Cloudflare?

AhrefsBot is on Cloudflare's verified bots list. In Cloudflare's firewall rules, create a custom rule that matches User Agent contains AhrefsBot, then set the action to Block or Challenge.

User-agent matching alone still leaves room for spoofing. Pair it with IP range matching using Cloudflare's IP Source field to cut down on false positives and false negatives.

On Cloudflare Pro and above, you can also manage AhrefsBot through the Bot Management dashboard, where it shows up as a verified bot alongside Googlebot and Bingbot.

Is AhrefsBot safe, or is it a malicious bot?

AhrefsBot is a legitimate crawler run by a reputable SEO company. It is Cloudflare Verified, respects robots.txt directives, follows crawl-delay settings (outside of JS rendering passes), and publishes IP ranges and verification methods publicly.

It isn't a malicious bot, a scraper, or a security threat. If you see aggressive traffic claiming to be AhrefsBot, validate the source IP with reverse DNS lookup and confirm it against Ahrefs' published IP ranges - impersonator bots exist, but real AhrefsBot traffic is safe and verifiable.

Does AhrefsBot respect robots.txt and crawl-delay?

Yes. AhrefsBot is documented to respect robots.txt directives, including Disallow and Crawl-delay.

JavaScript rendering is the exception for crawl-delay. During JS-heavy renders, AhrefsBot pulls multiple assets at once to mirror browser behavior, and crawl-delay doesn't apply to those parallel fetches.

If you run a single-page application or a JS-heavy framework, plan for that behavior and back up robots.txt crawl-delay with server-side rate limiting when you need strict request throttling.