What is AhrefsBot, and how do you check, slow or block it?

AhrefsBot is the web crawler run by Ahrefs, the SEO tool company. It visits pages across the web and follows the links on them, and what it collects fills the link database behind Ahrefs' reports. In a server log its name, AhrefsBot, sits inside the user-agent string.

This page gives the exact user-agent string, the second Ahrefs crawler, how to check that a hit is from Ahrefs, how to slow the bot or block it in robots.txt, and what blocking changes in Ahrefs' own data. If crawling itself is new ground, how a search engine works covers the basics.

What AhrefsBot is and what its user agent looks like

Ahrefs' robot page states the crawler's purpose in one line: "Powers the database for both Ahrefs, a marketing intelligence platform, and Yep, an independent, privacy-focused search engine." When the bot reads a page, the links on that page can flow into Ahrefs' index.

The full user-agent string, as Ahrefs prints it on that page, is:

Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

Two parts of it matter. AhrefsBot/7.0 is the name and version, and http://ahrefs.com/robot/ is the address of Ahrefs' own page about its crawlers. That address is why people who see the string in a log search for "ahrefs com robot": the crawler is pointing to its own documentation. The token to use in robots.txt rules is AhrefsBot, exactly as it appears before the slash.

On size, Ahrefs' glossary says: "Every minute, AhrefsBot crawls 5 million pages." That is Ahrefs' own figure, printed on its own page.

On how the bot moves, Ahrefs' Help Center (article dated 3 September 2026) says the crawler does not generate URLs of its own: it only follows links it finds on the web, and it always uses GET requests. The same article says the bot re-crawls a site from time to time to check the status of links it found before.

Ahrefs is not the only tool running a crawler. Majestic runs its own for the same purpose, and we compare the two tools in Ahrefs and Majestic.

AhrefsBot and AhrefsSiteAudit: the two Ahrefs crawlers

Ahrefs runs two crawlers, and they do different jobs. AhrefsBot crawls the web on its own and builds the link database. AhrefsSiteAudit crawls a site when someone audits that site in Ahrefs' Site Audit tool. A hit from the first is the public crawl passing through. A hit from the second means someone has set up an audit of your site.

The desktop user-agent string for AhrefsSiteAudit is:

Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)

Ahrefs prints a mobile string for it as well.

On speed, the robot page says AhrefsSiteAudit limits its crawling to 30 URLs per minute at most, to keep the load on website servers down. Ahrefs' Help Center adds that a verified site owner can raise that limit for their own site.

On robots.txt, AhrefsSiteAudit obeys it by default, and a verified site owner can ask it to ignore robots.txt on their own site.

For both crawlers, the robot page says: "Both bots strictly respect robots.txt on both disallow and allow rules, as well as crawl-delay directives."

How to check that a hit is from AhrefsBot

A user-agent string can be typed by anyone, so the string alone does not identify the visitor. Ahrefs publishes two things to check a hit against.

The first is the hostname. The robot page says: "The reverse DNS suffix of the IPs hostname is always ahrefs.com or ahrefs.net." So a reverse DNS lookup on the IP address in the log should return a hostname ending in one of those two.

The second is the IP lists. Ahrefs says it crawls from IP ranges it publishes. The Help Center article "What is the list of your IP ranges?" (dated 3 September 2026) carries the list, and Ahrefs also serves it as JSON at two addresses:

https://api.ahrefs.com/v3/public/crawler-ip-ranges

https://api.ahrefs.com/v3/public/crawler-ips

The same article lists addresses Ahrefs has retired, so pull the list fresh rather than keeping an old copy.

For the lookup itself, Google documents a manual check for its own crawlers (page last updated 20 March 2026): run a reverse DNS lookup on the IP, then a forward DNS lookup on the hostname returned, and confirm it resolves to the same IP. For AhrefsBot, check the hostname against Ahrefs' sentence above and the IP against Ahrefs' published list.

One more signal: Ahrefs' robot page says: "Both AhrefsBot and AhrefsSiteAudit are recognized as verified "good" bots by Cloudflare, a leading web security and performance company." Cloudflare's verified bots page (last updated 1 July 2026) lists the conditions a bot must meet for that list; one is that it obeys robots.txt and crawl directives and keeps reasonable request rates.

How to slow AhrefsBot down

Robots.txt can slow AhrefsBot without blocking it. The record, as Ahrefs prints it, is two lines:

User-agent: AhrefsBot

Crawl-Delay: [value]

Ahrefs' note with the record: the value is the time in seconds between two consecutive requests. Replace [value] with the number of seconds you want. The bot keeps visiting, but waits that long between one request and the next. The record sits in the same robots.txt file as any disallow rules, under the bot's own user-agent token.

One limit applies, in Ahrefs' words: "Crawl-delay is strictly followed when requesting HTML pages, ensuring that we do not exceed the specified rate limits. However, it cannot be followed when rendering JavaScript."

Status codes also shape the crawl. Ahrefs' robot page says: "If we encounter non-200 status codes, especially 4xx or 5xx errors, we automatically reduce our crawling speed for that site." The same page says a site can return 4xx or 5xx codes for the duration of an outage or maintenance window, and the bot will detect the errors and back off on its own. Our guide to HTTP status codes explains what each code means.

Caching cuts the load as well. During crawls, Ahrefs says it caches frequently requested files such as images, CSS and JavaScript, so repeat visits fetch less and bandwidth use drops.

How much crawling a site can take, from any crawler, is the subject of crawl budget.

How to block AhrefsBot

To stop AhrefsBot crawling certain pages, such as one section of a site, Ahrefs prints:

User-agent: AhrefsBot

Disallow: /path-to-disallow/

To block the whole site, the Help Center (article dated 3 September 2026) prints:

User-agent: AhrefsBot

Disallow: /

The robot page says the same Disallow directives stop AhrefsSiteAudit, with its own token, AhrefsSiteAudit, in the user-agent line. The change is not instant. Ahrefs says: "Please note that AhrefsBot may need some time to pick up the changes in your robots.txt file. This is done before the next scheduled crawl."

To let the bot back in, the Help Center says to remove the two blocking lines. Where the bot cannot read a site's robots.txt file at all, the same article prints an allow record:

User-agent: AhrefsBot

Allow: /

The same article names two error codes that point to a block somewhere other than robots.txt: a 403 from an IP block in the web server configuration, and a 406 from a firewall or security plugin blocking by user agent.

Blocking can also happen at the firewall. On Cloudflare, custom rules live in the dashboard under Security rules: create a custom rule, match the user agent field with the operator "contains" and the value AhrefsBot, and choose the action Block (Cloudflare's developer docs; the custom rules page was last updated 3 August 2026). Cloudflare states that its separate verified bot field requires an Enterprise plan with Bot Management enabled.

What blocking changes in Ahrefs' data, and why a new link can take time to show

This is the part that matters to anyone who builds or buys links.

What blocking changes, in Ahrefs' own words: Patrick Stox wrote on the Ahrefs blog on 22 May 2025: "Blocking these bots will mostly impact the link index of the tools." And: "It doesn’t matter for traffic estimates, keyword rankings, top pages, etc. Those are built from different data sources." The same post says that, for Ahrefs, blocking also affects the internal links it shows and its page history feature, which shows changes to a site's pages over time.

The mechanics run in two directions. A site that blocks AhrefsBot stops Ahrefs reading that site's pages, so the links on those pages, the links that site gives to others, are not read. Links pointing to the blocked site from other sites can still be found when they sit on pages Ahrefs has crawled. Ahrefs' Help Center (3 September 2026) says: "Note: Backlinks reports will be available if we crawled other sites that have backlinks to the target site."

How common is naming the bot in robots.txt? The HTTP Archive's Web Almanac 2025 (SEO chapter, published 15 January 2026) found: "ahrefsbot appears in 9.29% of desktop sites and 9.50% of mobile sites." That counts robots.txt files that name the bot, not crawl volume.

A related question: why can a new link be missing from Ahrefs for a while? Ahrefs' Help Center (3 September 2026) answers: "So the short answer is that it can take a few days up to some weeks before we crawl your new backlink." The same article lists some of what its crawl order weighs: crawl priority based on DR, crawl speed for a page based on UR, the number of pages on a site and the number of new backlinks found. On the database as a whole, another Help Center article from the same date says a complete update of the whole backlink database takes about two months.

Where each tool's link count comes from is covered in how many backlinks you have, and DR is explained in our guide to Domain Rating.

Can you ask Ahrefs to crawl a page? Not for Site Explorer or Content Explorer: its Help Center says you cannot submit links, URLs or an XML sitemap, or ping Ahrefs, for an on-demand crawl.

One site's logs: how often AhrefsBot visits

QED Web Design, a web design business in Devon, published its own server log counts on 14 November 2025. For October 2025 it counted 4,676 requests from AhrefsBot, 870 from SemrushBot and 289 from DotBot, and wrote: "AhrefsBot alone issued nearly as many requests as Google."

It also gave a timeline: it blocked AhrefsBot in its robots.txt file on 1 November, and the bot crawled again on 2 November.

Set beside that is Ahrefs' own statement on its robot page: the bot may need some time to pick up robots.txt changes, and that happens before the next scheduled crawl.

These are the counts of one site in one month. The counts on another site will differ.

What this means for links we place

Ahrefs lists a new link only after AhrefsBot has crawled the page it sits on, so a new placement can be live on a published page before it shows in Ahrefs.

What we check ourselves: after publication, our QA team reviews the live page. The page must be live, the link must sit in the body copy, the anchor and target URL must match the order, and the page must be indexable. Only then is the URL reported to your dashboard. That is our post-publication check.

So the record of a delivered link is the live URL in the report, which you can open and read. Getting host pages indexed by search engines is a separate subject, covered in indexing backlinks.

Frequently asked questions

What is the Ahrefs bot called?

The Ahrefs bot's name is AhrefsBot, and AhrefsBot is also the token to use in robots.txt rules. Ahrefs runs a second crawler, AhrefsSiteAudit, which crawls a site when someone audits it in Ahrefs' Site Audit tool. Both identify themselves by name in server logs.

What is the AhrefsBot user agent?

The full string, as Ahrefs prints it, is Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/). The address at the end is Ahrefs' page about its crawlers. AhrefsSiteAudit uses a different string: Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit).

What are the Ahrefs IP addresses?

Ahrefs publishes its crawl IP ranges in its Help Center and as JSON at https://api.ahrefs.com/v3/public/crawler-ip-ranges. Hostnames for the crawler end in ahrefs.com or ahrefs.net. The list changes as Ahrefs retires old addresses, so pull it fresh rather than keeping a copy.

How do I block AhrefsBot in robots.txt?

Add two lines: User-agent: AhrefsBot and Disallow: /. To block one section instead, put its path in place of the slash. Ahrefs says the change is picked up before the next scheduled crawl, so it is not instant.

How do I limit how fast AhrefsBot crawls?

Use the Crawl-Delay line under User-agent: AhrefsBot, with a number of seconds as the value. Ahrefs says the delay is followed for HTML pages but cannot be followed when rendering JavaScript.

How do I get Ahrefs to crawl my site?

You cannot force a crawl: Ahrefs takes no manual submissions for Site Explorer, and it finds pages by following the links it comes across. Its Help Center says a new domain may take one to two weeks to be picked up. Check that your robots.txt allows AhrefsBot; the status check on Ahrefs' robot page shows whether its bots can crawl your site.