How Often Does ChatGPT Change Its Recommendations? What Our Tests Show
Asked the same Dallas IT question three times, ChatGPT named seven different MSPs. Five runs of another question gave five different shortlists. What moved, what held, and how to measure it.
How often does ChatGPT change its recommendations? Every time you ask, it can, even minutes apart. In our tests, the same question asked three times named seven different MSPs, and five runs of another question gave five different shortlists. A few names held steady. Most came and went with the pages ChatGPT happened to read. So measure how often you’re named across many runs, not once.
Founder, Aquinel · Observed · 8 min read
Note on verification: The main numbers in this post come from three of Aquinel’s own repeat-run ChatGPT tests, run with web search on between August and October 2026. They are small tests, not peer reviewed. We name the agencies from our own category test because we published that test in full. We hide the names of the MSPs from the Dallas test. Two outside studies appear near the end. SparkToro sells software, so we label its work as vendor research.
You run an MSP business, a managed service provider (MSP). You ask ChatGPT who the best IT company in your city is. It names you. You take a screenshot and feel good.
Ask again tomorrow. You might not be there.
Since August 2026, we’ve put the same buyer questions to ChatGPT again and again, and written down every answer.
How often does ChatGPT change its recommendations?
It can change them between two answers asked minutes apart.
On 24 October 2026, we asked ChatGPT one question three times in a row: “Who is a good managed IT services company for a 30-person law firm in Dallas?” We used the same words, on the same day, from the same US location. Each answer named three or four MSPs. Across the three answers, ChatGPT named seven different MSPs.
| MSP (names hidden) | Answers that named it |
|---|---|
| MSP A | 3 of 3, listed first each time |
| MSP B | 2 of 3 |
| MSP C | 2 of 3 |
| MSP D | 1 of 3 |
| MSP E | 1 of 3 |
| MSP F | 1 of 3 |
| MSP G | 1 of 3 |
Source: Aquinel three-run test, ChatGPT with web search on, US location, 24 October 2026.
MSP A held its spot. MSPs D to G each appeared in one answer and missed the other two. Any of those four would see a very different picture depending on which answer they happened to check.
Did we see the same thing in other tests?
All three of our repeat-run tests show the same pattern.
On 17 September 2026, we tested our own category. We asked ChatGPT “best ai seo agency for MSPs” five times. We used a fresh temporary chat each time, with personalization off.
| What we counted | Result |
|---|---|
| Agencies named across five runs | 12 |
| Agencies named in all five runs | 0 |
| Most consistent agency (Jumpfactor) | 4 of 5 runs |
| Identical shortlists | None. All five were different. |
| Our own result | Named in 2 of 5, last both times |
Source: Aquinel five-run test, 17 September 2026.
Twelve agencies shared five shortlists. The full log is in our five-run case study.
We threw out one extra run from that test. Before that run, the chat’s personalization setting had switched back on. That run put us first out of seven. No clean run ever did.
Does the wording change the answer too?
On 29 and 30 August 2026, we ran 18 ChatGPT queries about GEO agencies in Chennai. We used 8 different phrasings. We were named in 4 of the 18.
All four came from one phrasing: “best GEO agency in Chennai for B2B SaaS”. That phrasing named us in 4 of its 7 runs. The near-identical wording “best geo agency for b2b saas chennai” named us in 0 of 5 runs.
The answer changes between runs, and it changes between wordings. Your buyers don’t all type the same words, so test a few wordings of each question.
Why does ChatGPT keep changing its recommendations?
In our tests, ChatGPT searched the web before naming companies. It read a handful of pages, then built its shortlist from what those pages said. Each run, it read a slightly different set.
The names followed the pages.
In our five-run test, we were named in runs 1 and 3. In both, ChatGPT read our own MSP service page. In the other three runs, it didn’t read that page, and it didn’t name us.
Runs 2 and 5 came almost entirely from best-of articles. ChatGPT didn’t open a single agency’s own page in either run. None of those articles listed us, so we had no way in.
The Dallas test looked the same. One “best IT providers for Dallas law firms” list appeared as a source in all three answers. ChatGPT also opened some MSPs’ own law firm pages, but not the same ones each time.
| Fairly steady across runs | Changed from run to run |
|---|---|
| One or two names at the top | Most of the other names |
| The best-of lists ChatGPT cited | Which MSP pages it opened |
| The general shape of the answer | How many names it gave |
What stays the same?
The top name held steadiest. MSP A came first in all three Dallas answers. Jumpfactor made four of our five shortlists.
The best-of lists held steady too. In our five-run test, one roundup page was cited in four of the five runs. In the Dallas test, one list was cited in all three. ChatGPT keeps going back to these lists. Being on them gives you more chances to be named. Without them, you depend on ChatGPT happening to open your own page.
What do outside studies say?
SparkToro published a study on 28 January 2026. Volunteers ran 12 prompts a combined 2,961 times across ChatGPT, Claude and Google’s AI. They found less than a 1 in 100 chance that two answers to the same question listed the same brands. SparkToro sells software, so treat this as vendor research.
Researchers at the University of St. Gallen tracked daily AI answers for six weeks across four engines. They published in April 2026. Only about 35% of cited sources overlapped from one day to the next. They saw similar churn when the same prompt ran at the same time.
The two studies disagree on how many runs you need:
| Source | Runs suggested |
|---|---|
| Aquinel (our working method) | 5 per question, repeated monthly |
| University of St. Gallen, April 2026 | At least 7 per prompt per day |
| SparkToro, January 2026 (vendor research) | 60 to 100 per prompt |
We can’t settle that. All three agree on one thing: a single check tells you very little.
How should you measure ChatGPT recommendations?
Count how often you’re named across runs. Here’s the method we use.
- Write ten questions your best buyers would type. Put an industry, a city and a company size in each one.
- Write two or three wordings for your most important questions.
- Open a temporary chat with personalization off. Turn web search on.
- Ask each question five times. Use a new chat every run.
- Write down every MSP named. Write down every page ChatGPT cites.
- Count your ratio for each question. For example, “named in 2 of 5”.
- Repeat once a month. Watch the trend over three months.
Expect noise. A change of a few points from one month to the next means little. A steady climb over three months means something real.
Look hardest at the sources you wrote down. Ask to be added to the lists ChatGPT cites, and fix the pages it opens. We cover what ChatGPT reads before it names an MSP in Does ChatGPT Recommend MSPs?
The bottom line
ChatGPT can change its recommendations between two answers. In our tests, one question named seven MSPs in three tries. Five runs of another question gave five different shortlists. One wording named us, and a near-identical one didn’t.
The useful number is how often ChatGPT names you out of ten runs, across a few wordings. Track that number, find the pages behind it, and get onto them. If you want help, see our services and SEO for MSPs.
Questions
How often does ChatGPT change its recommendations?
It can change them on every run. On 24 October 2026 we asked ChatGPT one Dallas law firm question three times. It named seven different MSPs. Only one MSP appeared in all three answers.
Does ChatGPT give the same shortlist if I ask the same question twice?
In our tests, it didn’t. On 17 September 2026, five runs of “best ai seo agency for MSPs” gave five different shortlists. Twelve agencies appeared in total. None appeared in all five runs.
Why does ChatGPT recommend different companies each time?
In our tests, ChatGPT searched the web before naming companies, and it read a slightly different set of pages each time. The names followed the pages. We were named only in the runs where ChatGPT read our own page.
Does the wording of the question change the recommendation?
Yes. In our August 2026 Chennai test, one phrasing named us in 4 of 7 runs. A near-identical rewording named us in 0 of 5 runs.
How many times should I ask ChatGPT to know if it recommends me?
More than once. We use five runs per question as a starting point and repeat monthly. Outside studies ask for more: University of St. Gallen researchers suggest at least seven runs a day, and SparkToro suggests 60 to 100 runs per prompt.
Sources
- Aquinel three-run test of “Who is a good managed IT services company for a 30-person law firm in Dallas?”, ChatGPT with web search on, US location, Aquinel, 24 October 2026. Internal data.
- We asked ChatGPT for the best AI SEO agency for MSPs five times (our own study), Aquinel, 17 September 2026. /case-studies/best-ai-seo-agency-for-msps-chatgpt-5-runs
- ChatGPT GEO agency Chennai, 18 runs (our own study), Aquinel, 29–30 August 2026. /case-studies/chatgpt-geo-agency-chennai-18-runs
- New research: AIs are highly inconsistent when recommending brands or products (vendor research), SparkToro, 28 January 2026. https://sparktoro.com/blog/new-research-ais-are-highly-inconsistent-when-recommending-brands-or-products-marketers-should-take-care-when-tracking-ai-visibility/
- Don’t Measure Once: Measuring Visibility in AI Search, Schulte et al., University of St. Gallen, 10 April 2026. https://arxiv.org/pdf/2604.07585
If you would like us to walk you through how this would work for your firm, the call is free.