Blog · 25 September 2026 · By HeardOf

AI crawlers take more than they send back, on the latest figures: crawl-to-referral ratios, compiled

Four publishers count AI crawler traffic on their own networks, two of them count what comes back, and a fifth site republishes one network's figures by the month. We read their pages on 25 September 2026 and put each figure beside the unit its source states, because the figures do not share one: crawl requests per referral on Cloudflare's network, AI bot visits per human referral on TollBit's, shares of bot traffic on Fastly's, requests per month on Vercel's. Nothing here is our measurement, no figure is compared across networks, and each carries its window and the date its page prints; the ones older than 90 days are marked historical. Cloudflare's live Radar page answered our fetch with a bot check and was not read, so Cloudflare's figures here are the ones its own posts print and the ones technologychecker.io says it pulled from Radar's API. What the crawlers are is in AI crawlers in one table, and what a CDN does to them is in is your CDN blocking AI crawlers?. This post is the volumes.

How many pages does an AI crawler fetch for every visitor it sends back?

For calendar July 2026, by technologychecker.io's reading of Cloudflare Radar's API pulled on 1 August 2026: Anthropic 1,917 crawl requests per referral, Perplexity 289, OpenAI 251, Microsoft 36 and Google 4.7. The unit is Cloudflare's. Its post of 1 July 2025, historical by our 90-day rule, defined the crawl-to-refer ratio as requests from a platform's user agents that returned HTML, divided by requests for HTML whose Referer header named one of that platform's hostnames, "always normalized to a single referral request", and added that traffic referred by Claude's native app "does not include a Referer: header", which it believes holds for other native apps too, so the ratios "may overstate" and "it is unclear by how much". technologychecker's robots.txt report, published 3 April 2026 and updated 3 September 2026, prints the table below as its own pull, says the figure is "crawl requests per referral, not a percentage", and declines to print Mistral's July value as a trend because, it says, the weekly series returned no referrals in four of July's five weeks. We did not repeat the queries; the rows are technologychecker's readings, not a Cloudflare page.

Cloudflare's own printed series is a year older. Its post of 29 August 2025, historical, gave monthly ratios for January to July 2025 as crawls per referred page visit on its network: Anthropic 286,930.1 in January and 38,065.7 in July, a change the post printed as −86.7%; Perplexity 54.6 and 194.8, printed as +256.7%; OpenAI 1,217.4 and 1,091.4; Microsoft 38.5 and 40.7; Google 3.8 and 5.4. technologychecker's table also prints a July 2025 column: its Anthropic, Perplexity, OpenAI, Microsoft, Yandex and Google values sit within 2% of the figures in Cloudflare's post for the same month, and its Baidu, ByteDance and DuckDuckGo values (0.85, 0.90 and 0.32) sit within 0.05 of Cloudflare's one-decimal figures (0.9, 0.9 and 0.3); both are historical, and the paragraph above prints Cloudflare's own.

Which AI crawler traffic statistics does each source publish, and in what unit?

More than three units, and a row in one unit cannot be ranked against a row in another: crawl requests per referral (Cloudflare's posts, technologychecker's readings), AI bot visits per human visitor referred from AI applications (TollBit), a share of traffic, requests or referrals (Fastly, the purpose splits, technologychecker's bot share, TollBit's 0.05%), a share of sites (Cloudflare's post of 15 September 2026), and requests per month on one host's network (Vercel). The table holds one row per figure with the unit its source states, the network, the window and the page date, and marks historical any figure whose page or window is more than 90 days old on 25 September 2026. AI crawler traffic statistics arrive in those shapes, and a post that puts them in one column has already made a claim none of the sources made.

Do AI crawlers send any traffic back at all?

Fewer visits than they take, on the two sources that count referrals on their own networks. TollBit's State of the Bots for the first two quarters of 2026, read 25 September 2026, says it takes "179 AI bot visits to get a single human visitor referral in return from AI applications for European publishers", "over 3x" the North American rate; that the ratio "went from 150:1 in Q1 2026 to 227:1 in Q2 2026", where the first-quarter figure is historical by our rule and the second-quarter window closed on 30 June 2026; and that "AI bot traffic accounted for only 0.05% of total human referrals on European sites", which "carry roughly 3x the AI load per human visitor as their North American peers, seeing about 1 AI bot scrape for every 33 human visits". Its sample is "publishers on TollBit", and it says the numbers "reflect only identified bots". The page prints no publication date, its foreword is dated in its own text to July 2026, and sections 2 to 4 of the report sit behind a form we did not fill, so the figures here are the ones its executive summary displays.

On Cloudflare's network the same question, read the other way in technologychecker's July 2026 table: Google's crawlers made 4.7 requests per referral and DuckDuckGo's 2.4, OpenAI's 251, Perplexity's 289 and Anthropic's 1,917, and the page says Mistral's sent no referrals in four of July's five weeks; what the OpenAI, Anthropic, Perplexity and Google user agents are for, in their owners' words, is in AI crawlers in one table, which does not cover DuckDuckGo or Mistral. Cloudflare's own post of 29 August 2025, historical, adds the search side for one cohort: in "our dataset of news-related customers", Google's referral traffic in March 2025 was 9% below January and April's 15% below.

How much of the crawling is for training rather than search?

On both windows we read, training's share of declared crawl purpose on Cloudflare's network is above search's. technologychecker.io's crawler-statistics page, updated 3 September 2026, says it re-queried Radar that day for August 2026 and prints, on its own bucket names, Training 40.15%, Mixed Purpose 36.96%, Search 16.45%, User Action 4.80% and Undeclared 1.65%. Cloudflare's own post of 29 August 2025, historical, gave 80% training, 18% search and 2% user actions "over the past 12 months", classified "based on operator disclosures and industry sources", and 72% training in July 2024 against 79% in July 2025.

Fastly cuts AI bot traffic on its own customers' applications by bot type rather than by declared purpose, so its split is a different unit on a different network: on its Next-Gen WAF and Bot Management customers, "over 130,000 applications and APIs", crawlers were 80% and fetchers 20% of AI bot traffic between 16 April and 15 July 2025, and 71% and 29% between 1 July and 30 September 2025, both windows historical. What site owners do about the training share is one sentence in Cloudflare's post of 15 September 2026: "less than 1% of Cloudflare sites choose to block Search bots" and "17% of sites choose to enable some mechanism to block training", with no window stated. Which Cloudflare setting does what is in is your CDN blocking AI crawlers?.

How are requests split between crawlers, on each page that counts them?

Each page prints its own split, on its own network and window, and the splits are not added here. Cloudflare's post of 29 August 2025, historical, put the AI-only bots on its network in July 2025 at GPTBot 28.1%, ClaudeBot 23.3%, Meta-ExternalAgent 17.7%, Amazonbot 14.1% and Bytespider 5.8%, and Googlebot at 39% of all AI and search crawler traffic that month. Fastly's report for 1 July to 30 September 2025, historical, lists Meta's AI crawler at 42.78%, ChatGPT-User at 19.41% and GoogleOther at 17.27% of the quarter's crawler and fetcher traffic, with Meta at 60% of AI crawler requests and ChatGPT at 68% of AI fetcher traffic; its report for 16 April to 15 July 2025, also historical, had Meta at 52%, Google at 23% and OpenAI at 20% of AI crawler request volume, ChatGPT-User and OAI-SearchBot at "nearly 98% of all fetcher requests", and Perplexity at 1.53%. Vercel's post of 17 December 2024, historical, counted fetches on its own network for a month it does not name: Googlebot 4.5 billion, GPTBot 569 million, Claude 370 million, AppleBot 314 million, PerplexityBot 24.4 million, with Microsoft Copilot excluded because "it lacks a unique user agent for tracking". technologychecker.io's bot-traffic page, updated 3 September 2026, says bots of all kinds were 35.42% of web traffic in August 2026 by its reading of Radar.

How was this read?

With one plain fetch per page on 25 September 2026 between 00:05 and 00:13 UTC, a Chrome 140 user-agent string and no JavaScript, from one machine in one location; the evening of 24 September where we sit. Cloudflare Radar's AI Insights page answered that fetch with HTTP 403 and a bot check, and was not read: no figure in this post comes from it. TollBit's page answered 200 and draws its report with JavaScript, so it was rendered once in headless Chrome 153 over the DevTools protocol at 00:11 UTC; sections 2 to 4 of the report are behind a form we did not fill. Fastly's report directory links PDF files, which we read as text; its 2026 report is behind a form at a separate address, also not filled. Nothing was bypassed.

The pages: Cloudflare's posts of 1 July 2025 and 29 August 2025, both historical, and 15 September 2026; TollBit's State of the Bots, which prints no date; technologychecker.io's robots.txt report, AI crawler statistics and bot traffic statistics, each updated 3 September 2026; Fastly's Q2 2025 and Q3 2025 Threat Insights Reports, historical, linked from its report directory; and Vercel's post of 17 December 2024, historical. technologychecker's pages carry a row of Explore with AI links; we read those as data and followed none.

Nothing in this post is our measurement, so there is no claim about our own system in it to check; the closing's one sentence about what HeardOf does comes from our own systems, per our post on which engines we check. Everything else is a quotation from a page linked above, with the window it covers and the date it prints, and a page can change the day after it is read.

The four engines we ask, and what changed when Google AI Overviews became one of them