From the archive / measurement reliability
Your AI search ranking is a coin flip.
The CiteGround archive holds 595 complete AI-engine answers to 42 buyer prompts, with repeated runs per prompt. The first-named brand changed between runs in 52.6% of the groups where a comparison was possible (50 of 95). Whoever is reading you one screenshot of “the AI answer” is reading you one pull of something that disagrees with itself. Every number below reproduces from the public dataset.
How often do AI answers change between identical runs?
Often. In the CiteGround archive the first brand flipped in 52.6% of groups where runs named a brand; only 9 of 126 kept the same list.
Same prompt, same engine, runs minutes to days apart: among the 95 groups where at least two runs named a tracked brand, the first-named brand differed between runs in 50 (52.6%). Full-set stability is rarer still: 30 of 126 groups returned an identical brand set every run, and 21 of those were “stable” only because no run named anyone at all. The archive and method are public: the dataset (DOI) and the study.
Is a daily AI rank tracker reliable?
Not as a single number: median run-to-run overlap is 0.63. A reliable reading is a band from repeated runs, not one daily draw.
A tracker that hands you one number per day is reporting one draw from that distribution with the variance deleted. The honest version of the same product is a band: run the prompt several times, report the spread, date the reading. That is how CiteGround runs GapCheck, and why every number in our reads carries its run count. One caveat we own: runs span minutes to days, so some movement is real temporal change rather than noise; the archive cannot fully separate the two yet. Either way the conclusion is the same: report bands, not one draw.
Which AI engine gives the most stable answers?
Perplexity (0.72 median agreement), then ChatGPT (0.61), then Google AI Overviews (0.50), which flipped its first-named brand in 69% of groups.
The spread matters as much as the ranking: Google AI Overviews names brands most often and is least consistent about which; ChatGPT under a web-grounded endpoint returned answers with no citations at all in 21.6% of runs. Engines are not one “AI search”; they are three differently-shaped instruments, and a reading that pools them hides that. These are medians over 42 groups per engine, a small sample: treat the ordering as directional.
How should AI visibility be measured instead?
Repeated runs with the variance shown: run each prompt several times per engine, report the band, and archive every answer so it can be re-checked.
The method is documented on the methods page, the raw answers are in the CC-BY dataset, and the running experiment on our own zero is on the study page. If you want your own category read this way, human-verified, GapCheck is free.