How I Check the Sentence Behind a Visibility Score

I would not let a visibility score move a budget until I had read the sentence it was counted from. A detector that marks two strings near each other as "brand presence" can score a prompt echo, a disclaimer, or a real citation as the same win. The score is a label. The sentence is the evidence.

The method is The Seven Metrics That Measure AI Search Visibility. That page is the deeper resource. This essay is how I would use it as a founder: I would not accept any of those metrics until a sample of the raw answers had been read, and until retrieval-backed runs were stored apart from runs with no search.

What our own panel counted, and what a read did to the count

We run a daily panel across six AI surfaces: 89 logged runs between June 24 and September 22, 2026, 77 registered queries, 17,430 engine-query observations. One surface is a control arm, a frozen model with search turned off. The tracker scored 151 of that arm's 2,894 usable observations as owned brand presence, and 147 of those 151 rested on a proximity rule alone. If the name and the category term sat within 80 characters, the answer counted as a mention. The full audit is How to Tell If Your AI Visibility Tool Is Counting False Brand Mentions.

All 151 of the wins on that no-retrieval arm came from exactly two questions, both of which put the name and the category term in the prompt. The model was not being asked an open category question and volunteering a company. It was being handed the strings, and the detector was scoring the echo. A branded prompt measures something. It does not measure discovery.

A guard went in on 2026-09-11. The win rate on that arm fell from 6.0% (148 of 2,474 observations) to 0.7% (3 of 420). We read all three survivors. Each was an improvised description phrased one word outside the guard, so the error rate on what was left was 100%, on three wins. That is a string matcher being asked to do an entity detector's job. I had been reading the dashboard for 79 days before the sentences were read in bulk. The instrument was ours. The miss was ours.

The other 75 queries on that arm produced zero no-retrieval wins. I keep that zero next to the 151 on purpose. Same arm, same period, two kinds of question. One number is what a proximity rule did with a prompt that already contained the answer's nouns. The other is what it did when the prompt did not hand those nouns over.

The split that the score had blended

Strip the detector and the remaining difference is whether the answer has a citation. Across five retrieval-enabled surfaces in the same panel, the identical detector rested only 0.4% of its wins on string proximity alone, against 97.4% on the arm with no citations. Perplexity, ChatGPT, and Gemini did not score a single win that way. The model did not become a different model between June and September. The product either fetched a page or it did not.

Anthropic's documentation is direct about a frozen model: a snapshot with a training cutoff, no live search, and no way to know what happened after that cutoff unless a tool call gives it one (Anthropic). Perplexity's help center says each claim in a retrieval-backed answer is expected to trace to a source it fetched (Perplexity). OpenAI's developer documentation draws the same line: a base model's knowledge is fixed at a training cutoff, and anything after that, or anything narrow in the training mix, depends on whether the product lets the model look it up (OpenAI). Google's guidance on AI Overviews describes the other side of the same split: the system selects and cites web content at answer time, which is a different pipeline from anything stored in a model's weights (Google).

I would fund the pipeline that cites. A training snapshot updates on the vendor's schedule. A page on the open web can be retrieved the day someone asks. The glossary is where I keep the names of those layers from sliding into each other, and Machine Relations is the order I would build them: a citable page before a score that claims the page worked.

The four checks I would run on any score

I would ask whether the surface retrieves or recites. A retrieval-enabled answer cites a URL. A no-retrieval answer draws on a frozen snapshot. A report that blends them into one visibility score hides the split that changed our count from a proximity win to a read of the sentence.

I would ask whether a "mention" requires a cited source, a checkable claim, or only two strings near each other. If it is the third, I would ask for the false-positive rate on the no-retrieval arm specifically. We did not know ours until we computed it. A vendor who cannot produce that rate has not run the test.

I would ask what fraction of wins come from questions that already contain the brand name. All 151 of our no-retrieval wins came from 2 of 77 queries, and both queries typed the name and the category into the prompt. A model completing a sentence it was handed is a different event from a model naming a company on an open question. I would publish those rates in separate columns. I would not average them.

I would read the answers. Not the label. The sentence. The only way to catch a proximity hit that does not support the claim I think I am measuring is to read a sample of transcripts before the aggregate is allowed near a decision. After the 2026-09-11 guard, the three remaining wins still failed that read. A tighter pattern is not a substitute for the sentence.

How I would sample a score before it reached a budget meeting

I would not read every transcript. I would read enough that a proximity rule cannot hide. On our panel the failure was concentrated, which makes the sample small and the lesson sharp. Every one of the 151 no-retrieval wins came from 2 of 77 queries. A random sample of the dashboard would have been full of those two queries if the dashboard was full of those wins. I would stratify on purpose: some runs where the prompt already contains the name, some where it does not, some with a cited URL, some without. If I only sample the wins, I will meet the echo. The echo is what the 80-character window was built to find.

The September 11 guard is the reason I would not stop at a vendor's patch notes. The win rate fell from 6.0% to 0.7%, and the three wins that remained were the same error in a shape the new patterns missed. The error rate on that remainder was 100%, on three wins. A patch that shrinks the numerator can look like quality. The rate on what survives is the number I would ask for. I would also ask for the 79 days, or whatever the equivalent is on a vendor's side: how long the score was live before anyone read the sentence. We had the answer text. We had scored it. The read came late. I would rather delay a decision by a week than manage a quarter to an uncorrected proximity count.

The retrieval split is the column structure I would require in the export. On five retrieval-enabled surfaces, 0.4% of wins rested on string proximity alone. On the arm with no citations, 97.4% did. Perplexity, ChatGPT, and Gemini contributed none of those proximity-only wins. A spreadsheet that averages those arms will report a company as "up" because the no-search arm echoed a branded prompt. The other 75 queries on the no-retrieval arm produced zero wins. I would print that zero next to the 151 so nobody briefs the 151 as discovery. Anthropic's description of a frozen model, a cutoff and no live search unless a tool call supplies it, is why that arm cannot be talked into being a market. Perplexity's requirement that a claim trace to a fetched source, OpenAI's split between a fixed base model and a product that can look something up, and Google's account of AI Overviews selecting web content at answer time are the four documents I would link in the footnote of the export. They are not our methodology. They are the vendors describing the pipelines our methodology had blended.

I would also separate the seven metrics on the measurement page into ones that can be computed from a URL and ones that can be computed from a string. A metric that can be gamed by typing our name into the prompt does not belong in the same cell as a metric that requires a cited page. The panel can keep running daily. The budget can only move on the second kind, after a read.

What I would do

I would keep the panel. I would change what it is allowed to decide. Branded queries stay in a recognition series. Open category queries, on retrieval-enabled surfaces, are the series I would use for a budget. A co-occurrence match with no cited URL would be dropped, not patched with another exclusion list and then trusted. On the surface that does not cite, 97.4% of our scored mentions rested on two strings sitting near each other. I will not spend against a number with that composition.

Scope: any score that claims "presence," "mention," or "share of voice" across more than one engine, especially if the export does not attach the answer text and the citation URLs. The seven metrics on the page I linked are useful once each metric has a sentence under it. They are not useful as a blended index of proximity hits.

What would change my mind: a detector, ours or a vendor's, that on a fresh window posts a no-retrieval co-occurrence-only share near the 0.4% we saw on the retrieval-enabled surfaces, with the raw sentences available, and with branded prompts excluded from the discovery number. If a score can survive a read of the transcript, I will let it move a plan. The three no-retrieval wins that survived the guard did not survive that read: all three were unanchored. I would rather publish that than manage the company to the uncorrected count.

FAQ

What should I open first when a dashboard says presence went up?

The answer text for a sample of the runs, plus the rule that created the point. On our panel the rule was an 80-character window. It marked 151 observations on a frozen, no-search surface, and all 151 came from two branded questions. A rise that lives entirely inside those questions is a change in the echo, not a change in discovery.

Are a frozen model and a search-enabled answer the same kind of evidence?

They are different pipelines. Anthropic describes a frozen model as a snapshot with a cutoff and no live search unless a tool call provides it. Perplexity expects a retrieval-backed claim to trace to a fetched source. In our panel, proximity-only wins were 97.4% of the scored mentions on the arm with no citations and 0.4% on five retrieval-enabled surfaces. I would store those arms in different columns.

Did the September 11 guard fix the measurement?

It changed the count. The win rate fell from 6.0% to 0.7%. The three wins that remained failed a read and were the same class of error, phrased past the new patterns. The error rate on those three was 100%. I would treat a guard as a reason to reread, not as a reason to trust the next aggregate. Branded queries are now out of every presence rate I would publish.

What would I ignore even if the number looks precise?

A single score that mixes engines, mixes branded and open prompts, and does not keep the sentence. The other 75 queries produced zero no-retrieval wins. That precision is about a detector on one arm. It is not a market rank, and I would not brief it as one.