Blog · 20 September 2026 · By HeardOf

The four engines we ask, and what changed when Google AI Overviews became one of them

Which AI engines does this tool actually check?

Four: ChatGPT, Perplexity, Gemini and Google AI Overviews, each asked the same 40 buyer prompts, for 160 answers per audit when all four answer — our own settings as of 23 September 2026. The fourth was switched on that day; until then it was three, and an earlier version of this post explained why.

An engine that answers fewer than 90% of its prompts in a run is marked as partly answered and leaves the headline — our own rule, read on 19 September 2026 — and the fourth engine is held to the same bar as the other three, as of 23 September 2026.

One detail about the ChatGPT row, because it moves the numbers, and it is our own setting as of 19 September 2026: every ChatGPT call goes out with reasoning effort set to low and verbosity left at the model's default; we do not ask for shorter answers, because shorter answers name fewer brands and the score would move for a reason that has nothing to do with you.

The prompts are 40 per run in a fixed mix — 15 "best X for Y" questions, 10 alternatives, 10 head-to-head comparisons, 5 educational — generated from your category and the competitors you name, and each is asked once per engine per UTC day, with the answer reused across runs within that day; both are our own rules as read on 19 September 2026, and the fourth engine follows them as of 23 September 2026. Four brands are scored per run, you and up to three competitors, from the same 19 September reading.

ChatGPT and Perplexity lean on different sources for the same prompts — we reported that on 4 September 2026 in how to get your SaaS cited by ChatGPT and Perplexity, from a study of 30 B2B software brands on two engines, ChatGPT and Perplexity, 25 prompts, run that same day, and the source-level count behind it, Perplexity's 391 citations against ChatGPT's 173 from the 20 prompts whose raw answers survive, is in who the AI actually cites. That is why both are on the list, and why our headline, by the same 19 September reading, is the mean of per-engine rates rather than a pooled count over all answers; the fourth engine enters that mean as one rate among four, as of 23 September 2026.

How do we read Google AI Overviews, and what counts as an answer?

Through a Google results page, not through an API of Google's. For each of the 40 prompts, SerpAPI — a third-party service — fetches the results page for that prompt as a United States, English search and returns what is on it; we keep the AI Overview block's text and the links it references, and that is the answer we score, our own setup as of 23 September 2026. The other three engines are asked through their vendors' APIs. This one is a page, read for us, and Google decides whether the page carries an overview at all.

Google does not show an AI Overview on every search, and how often it shows one for the kind of prompt we ask — a bottom-of-funnel question about a software category — is the number that governs everything below. As of 22 September 2026 we found no published rate for that kind of question; the one we found for B2B technology queries in general rose from 36% to 82% between February 2025 and February 2026, as reported by ALM Corp on 3 March 2026, historical, which attributes it to BrightEdge; we did not find it on BrightEdge's own pages, and it does not separate buying questions from the rest. (Corrected 25 September 2026: this sentence first called the figure BrightEdge's own.)

When Google shows no AI Overview for one of the 40 prompts, that prompt is not dropped and the call is not marked failed: it counts as an answer in which nobody was named — not you, not any competitor — our own rule as of 23 September 2026, chosen over a third state between named and not named. Every prompt Google declines to answer with an overview therefore lowers AI Overviews' rate for every brand, and the headline through it. The rule rests on overviews appearing on most of the prompts we ask, and on Google not switching them on and off between weeks, so those two rates matter more than the rule does.

The same rule reaches two other places a subscriber reads, as of 23 September 2026: the report's list of what to fix can carry, for one prompt, an item saying nobody was named on one of the four engines beside an item saying a competitor was named where you were not — the first is the overview's empty page, and the two do not disagree; and a Monitor digest can report a name as dropped, or as entered, when Google shows an overview one week and none the next, or the reverse.

A page we could not read is the opposite case, and we keep the two apart, as of 23 September 2026: an overview that failed to load, a page we could not fetch, a fetch that timed out — each is a failed call, retried when a retry can help, and a prompt whose retries all fail is a failure for that engine, not an answer with nobody in it. Enough of them in one run and the engine's row in the report reads "Partly answered" or "No answers", and it leaves the headline under the 90% rule above. The line between the two cases is the fetch reporting an error: a page that comes back whole but without its overview is scored as nobody named, and we cannot tell that from Google withholding one — as of 23 September 2026.

What we do not see, also as of 23 September 2026: the overview a signed-in buyer sees; an overview in another country or language, since the page is fetched as a United States, English search, the country our ChatGPT and Perplexity calls are pinned to; and Google AI Mode, a separate surface that is not on the list. Google's own report — Search Console's AI-feature insights, rolled out globally on 31 August 2026 — tells you which of your own pages appeared, by country; it is not the answer with your competitors in it, which is why we read the page.

What happened to my scores from before the fourth engine?

Nothing, and nothing will. A score from before 23 September 2026 is the mean of three per-engine rates when all three answer and one from after it averages four when all four answer; they were not recomputed, they will not be, and the weekly comparison does not present the step between them as movement, though the twelve-week chart still draws both on one line — our own decision as of 23 September 2026. A score over an engine nobody asked cannot be recomputed: no answer was ever collected to recompute it from.

For Monitor subscribers that shows up as of 23 September 2026: the first weekly run after the switch is not compared with the week before — the dashboard says the two runs did not answer on the same engines and names Google AI Overviews as the one that answered in one run and not the other — and the comparison comes back the week after, when both runs are on four. The same happens in any later week an engine, this one included, fails to answer.

One correction we owe. Until 23 September 2026 the FAQ on our homepage said that when we added Google AI Overviews, past scores would get recomputed and you would get an email. Neither happened, the entry is gone, and we would rather say so here than let a FAQ change quietly.

Which competing tools say they check AI Overviews?

Fifteen of the nineteen we track, and since 23 September 2026 so do we. Two do not list it — HubSpot's AI Search Grader and Similarweb — and two, Bluefish and Conductor, we could not confirm either way: Bluefish names no engines on its own site, and our review could not confirm Conductor's coverage of AI Overviews.

The engine column comes from a competitor review dated 5 September 2026; where a vendor's own site named nothing, the table says so. That was before the block, and we have not re-read the vendors since; our own row is the only one dated after it.

What did Google do to scrapers in September, and does it reach us now?

From around 13 September 2026 Google may have become more successful at blocking scrapers and tracking tools, and vendors that collect Google results pages reported problems. Nozzle's Derek Perkins said he saw about an 80% drop in the data he could collect, and Sistrix's status page noted on 16 September that Google "has made further changes to the way search results are delivered", according to Barry Schwartz's report at Search Engine Roundtable on 18 September 2026.

It was not the first move. Between 23 June and 26 August 2026 Google rewrote result links into a google.com/goto redirect and blocked HEAD requests; Derek Perkins of Nozzle put the cost of resolving a five-page ranking at "between 500 and 1,000 requests" (ppc.land). The September block is the step after that.

Not everyone was hit the same way. Glenn Gabe's read, quoted in the same Search Engine Roundtable piece: "ahrefs still scraping ok, Semrush ok but slower, Sistrix way behind". Two of those three also sell AI-visibility measurement and appear in the table above, and both list AI Overviews.

Why a buyer of AI-visibility measurement should care: an AI Overview is a block on a Google results page. The usage figure Google gave at I/O on 19 May 2026 — AI Mode "has surpassed one billion monthly users", in Elizabeth Reid's post — is AI Mode's, a separate surface. Google's own figure for AI Overviews is over 2.5 billion monthly active users, in its post of 3 June 2026, as updated on 31 August 2026. A tool that reports AI Overviews visibility is, as far as we can tell, reading Google results pages to do it — and since 23 September 2026, so are we, through SerpAPI. If one collector saw about an 80% drop in the data he could get from Google in mid-September, by his own account, what happened to everyone's AI Overviews data is a real question, not a rhetorical one.

Until 23 September 2026 we could ask that from outside: our collection read no Google results page, and an earlier version of this post said so. Since then it is our question too, so here is what we can say from inside it. Google AI Overviews was switched on in production on 23 September 2026, the day this post was changed, and nothing here reports an answer it gave in production — from our own systems, read on 23 September 2026. Whether Google showed an overview on each of those pages is a separate number, and a different question from whether the page arrived. A block that reached our fetch would show up, as of 23 September 2026, as failed calls on that engine's row whenever the fetch reports an error; a page that arrives without its overview is scored as nobody named, and we cannot tell that from Google withholding one. That is the distinction we would want spelled out by any of the fifteen, so we are spelling it out for ourselves first.

As of 19 September 2026 we had seen no statement from any of the fifteen on what the block did to their AI Overviews collection, and this post does not claim to have looked again. What would count as an answer is short: a dated statement from each vendor of how much AI Overviews data it collected in the week before 13 September and the week after, in its own words. Gabe's note covers two of the fifteen — Ahrefs "still scraping ok", Semrush "ok but slower" — but his subject was Google results collection in general, not the AI Overviews block specifically. We are treating the two as one pipeline because the block sits on the same page as the ranking; that is our assumption, not his statement. For the other thirteen, the reporting we read names nothing either way. Two things we are not claiming. We are not claiming any vendor's numbers are wrong: a drop in collection can show up as a gap in a time series rather than a wrong figure, and only the vendor knows which. And we are not claiming ours are safer because they are newer. On 19 September we called the missing engine a real gap for anyone who sells to people who search Google; it was, and that gap is closed; the question this section asks is now asked of us as well.

What else do we not measure?

Four things, as our systems stood on 23 September 2026 and, where the fourth engine did not touch them, on 19 September 2026: the free check is not the audit, we hold one answer per engine per prompt, we do not record position, and we score only the brands you name. The engine list should not be the only limitation you hear about from us.

The free check on our homepage is not the audit. It asks one engine, Gemini, three prompts, about one domain, with no competitors — its settings as read on 19 September 2026, and adding the fourth engine to the audit did not add it to the check, as of 23 September 2026. It exists to show the shape of an answer in seconds, and it will not agree with the audit's 160 answers, because it is a different engine count and a different prompt count.

We hold one answer per engine per prompt, and we report rates — for each engine, the share of its answers that named or cited you, with a headline that is the mean of those per-engine rates; both are our own rules as read on 19 September 2026. A published study of exactly this measurement — Schulte, Bleeker and Kaufmann, arXiv, 8 April 2026 — puts the standard error of a single run at 0.370 and recommends at least 7 runs per prompt per day for brand-visibility monitoring. One answer per prompt estimates a rate without bias, so that is what we report, and we do not tell you "you are absent for this prompt", because at one answer that claim is close to a coin flip. That study is one of six we put side by side, each in its own unit, in how much AI answers change from one run to the next.

We do not record where in the answer you were named: as of 19 September 2026 our scoring stores whether you were named or cited, not where, and the fourth engine's overview is stored the same way as of 23 September 2026. An earlier version of the tool did record position; the product dropped it, and as of that date we have not put it back. SparkToro and Gumshoe's study of 2,961 runs — Rand Fishkin and Patrick O'Donnell, published 28 January 2026 — found less than a 1-in-100 chance that ChatGPT or Google's AI, asked the same prompt 100 times, returns the same list of brands in any two answers; the study's judgement was that "any tool that gives a 'ranking position in AI' is full of baloney", and we have no measurement that contradicts it.

And we score four brands per run — you and up to three competitors that you name — while brands the engine mentioned that you did not name sit in the stored answer text and are not counted; that is how our scoring stood on 19 September 2026. If a fifth competitor is the one eating your answers, the audit will not tell you that. Whether any engine reads your site's llms.txt is a different question, and not one the audit answers either; what the three vendors who wrote about that file serve on their own sites, measured with curl on 23 September 2026, is in does llms.txt actually work.

The list also still has no entry for Google AI Mode, Copilot, Claude, Grok, DeepSeek, Meta or Mistral — engines that several tools in the table above list. AI Mode is a separate Google surface, and adding AI Overviews did not add it. That is not a roadmap statement; it is a description of what we ran on 23 September 2026. What Google's own pages say separates AI Mode from AI Overviews, and where the Gemini app fits, is in AI Mode vs AI Overviews.

How this was read

Every claim this post makes about our own system carries one of two dates in the sentence that makes it: 19 September 2026, when we last read the settings the fourth engine did not change — the model behind each of the other three engines, the prompt mix, the once-a-day rule, the four brands, the free check — and 23 September 2026, the day Google AI Overviews was switched on, which dates everything about that engine: what it is read from, what it counts as an answer, what happened to earlier scores. Both halves come from our own systems and neither can be checked by a reader; we are saying so rather than letting you find the seam. Everything else carries a link or a date you can follow. The vendor engine lists come from a competitor review dated 5 September 2026 and have not been re-read since; where a vendor's own page named nothing, the table says so. The scraper block is reported by Barry Schwartz at Search Engine Roundtable on 18 September 2026, citing Nozzle, Sistrix and Glenn Gabe; the June–August link rewriting by ppc.land, citing Derek Perkins of Nozzle; the Search Console AI-feature insights by Google's own announcement; the AI Mode figure, which is AI Mode's and not AI Overviews', by Google's I/O post of 19 May 2026; the B2B technology overview rate by ALM Corp on 3 March 2026, citing BrightEdge. Each of those five is linked where it is used. Where we did not check, the text says so.

AI Mode vs AI Overviews, and where Gemini fits: Google's AI surfaces in Google's own words