Ask the same question twice. Get two different answers.
Checking what an AI assistant tells your customer looks simple: ask, then read the reply. But an assistant writes a fresh answer every time, shaped by chance, by the exact words, by which assistant it is, and by habits that have nothing to do with your product.
- Skelvane Kids SPF50+
- Solquin Little Ones SPF50
- Paleshore Mineral Kids SPF50+
The same question, asked 60 times. Who it named:
An illustration with invented brands. Your report shows this for your customer, the brands named instead of you, and each assistant.
Every answer is a fresh draw
An assistant does not look up a stored reply. It writes a new one each time, picking its words partly by chance. So one answer is a poll of one person. If an assistant names you in one answer in five, a single check misses you four times out of five, and the next check may say the opposite.
Asked “What should I make for dinner tonight?” 200 times, one AI model gave its most common dish only 15 times.
Our dinner study, 11 June 2026, OpenAI's model through its developer interface.
What we do: ask every question many times and report counts, such as "named in 9 of 60 answers".
Ask it another way and the list changes
People ask an assistant in many styles: a few search words, a plain question, their situation first, a request for one recommendation, a request for options to compare. Each style can bring up a different set of businesses.
| Search words | kimberley small ship cruise broome | 3 of 6 | |
|---|---|---|---|
| Plain question | Which small-ship cruises go to the Kimberley? | 1 of 6 | |
| Situation first | We're in our sixties and want to see the Kimberley without roughing it. What should we book? | 0 of 6 | |
| One recommendation | Can you recommend one Kimberley cruise? | 2 of 6 | |
| Options to compare | What are my options for a small-ship Kimberley cruise, and how do they compare? | 1 of 6 |
Illustration with invented results: the same brand, named in 7 of 30 answers overall, but anywhere from 0 to 3 of 6 depending on the wording.
What we do: turn your description of your customer into several situations, and ask each one in the styles people use. The questions about what to buy never name you.
Each assistant has its own favourites
Every assistant leans its own way, so which one your customer opens decides what they hear. In one of our studies we asked four AI models the same everyday question, 200 times each, in 20 different wordings.
"What should I make for dinner tonight?" · 200 answers per model, by main ingredient
Our dinner study, 11 June 2026. Each model asked 20 wordings of the question, 10 times each, through its developer interface.
What we do: test each assistant on its own, at a named model version, with the same questions.
When products look alike, the order decides
Assistants have habits that have nothing to do with the products in front of them. The strongest is position: when two options give it nothing to choose between, it takes whichever comes first. Give it a real reason, even one sentence that matters to the customer, and the habit disappears. So a "win" on a thin listing can be nothing more than going first.
With two identical products under invented names, assistants picked whichever was listed first in 96 of 100 decisions, so every comparison we run is asked in both orders.
Our study of five well-known brands, May 2026, OpenAI's and Google's models through their developer interfaces.
What we do: when we put products side by side, in a Full picture, we swap the order on every comparison and add hidden checks that show whether the assistant is reading. At every depth we also ask about a brand that does not exist, so you can see how sure the assistant sounds about a name it has never seen.
The same product wins or loses depending on who is asking
Assistants match a product to a need. One sentence on a listing that speaks to your customer's situation can turn a loss into a win, and a count of mentions will not show it.
In one of our studies, an assistant chose between five options for 23 kinds of customer, 200 times each. In 59% of cases it chose the same way at least nine times in ten. Asking again added little; asking as another kind of customer changed the answer.
Our study of expedition cruises, May 2026, three AI models through their developer interfaces.
What we do: take your customers from your brief, beside the standard customers for your category, and ask as each one, including before they know what to buy, from a clean start. We also ask for just one, so you see who it picks. In a Full picture, we test the decision itself: your product beside your rivals, as each is listed today.
The assistant you test today is not the one you test next month
Providers update and retire their models in the background. OpenAI, for example, retires its original GPT-5 versions on 11 December 2026. The same question can get a different answer after an update, with nothing in the app to say so.
What we do: treat every report as a dated measurement. The versions and dates are on the cover, the settings are held fixed, every figure comes with its range, and the main comparisons say how sure they are: clear, indicative or too close to call.
On every report cover
Example
- Measured
- 2 December 2026, 9.10am to 10.40pm, Sydney time
- ChatGPT
- the model behind it, version gpt-5.6-luna, through OpenAI's developer interface
- Gemini
- the model behind it, version gemini-3.5-flash-lite, through Google's developer interface
- Market
- Australia
- Answers
- 1,230, each one kept
Designed before it runs, checked after
Designed before it runs
Every test is written down, with what we expect to find, before a single question is sent. It then passes an independent design review, and a design that fails is changed before it runs.
Checked by someone other than the assistant
A First look lists every point the assistant makes about you, for you to read against your own page. In a Full picture, each statement is checked against the facts you confirm, by two judges from AI families that are not under test, and a person reads every statement we print as wrong.
What we test
The models behind ChatGPT, Gemini, Claude, Grok, DeepSeek and other assistants, through the developer access their makers offer. This is how a question can be asked thousands of times under the same conditions.
Our studies have already compared four makers' models side by side. Your proposal names the assistants in your report.
The ChatGPT and Gemini apps add memory, location and settings of their own, so the app can give one person a different answer. Every report says which models were tested and when.
What every report shows
- The brief as you confirmed it: your customer and their situations
- The model versions and the dates of the test
- The number of answers behind every result
- What the assistant says about a brand that does not exist, as the baseline
- Each figure with its range; the main comparisons labelled
So we measure it the way consumer research is done
So we ask many times and count.
Chance
Every question asked many times. Every result is a count.
Wording
Each of your customer's situations, asked in the styles people use.
Which assistant
Each assistant on its own, at a named version.
Habits
The order swapped on every comparison, and a made-up brand as the baseline.
Fit
Asked as each of your customers, with the single pick.
Change
Every report dated, with the versions on the cover.