Does AI recommend the same products when you repeat a question? Check this before basing a visibility report on one answer. In an arXiv preprint with no reported peer review, researchers found that recommended products and displayed sources varied across repetitions, while responses obtained through an API, a software connection to the system, differed from the consumer interface. A preprint shares research ahead of peer-reviewed publication. DijitalPi commentary: Record which app supplied the answer and whether you repeated the question. The study measured neither sales nor clicks nor advertising effects. Predominantly English queries collected from the Netherlands do not establish validity for Turkish or Türkiye.
ACADEMIC WORK REVIEWED
"If I Had to Buy Just ONE: Galaxy S26 Ultra": Auditing AI-Generated Product Recommendations
ResearchersLucas G. Uberti-Bona Marin, Thales Bertaglia, Giovanni Astante, Bram Rijsbosch, Gijs van Dijck, Anikó Hannák, Gerasimos Spanakis, Konrad Kollnig
arXiv · 16 September 2026 · v1, no peer review reported · Preprint · Source language: English
English review: 17 October 2026 · The source date appears in the citation above.
Editorial review draft
The source has been checked; the review still carries its original editorial status. DijitalPi did not conduct a customer experiment for this paper.
LET'S READ THE RESEARCH TOGETHER
We first explain the researchers' question, method and findings. We then discuss how to interpret the results, clearly separating DijitalPi's commentary from the source.
01 / WHAT DID THE RESEARCHERS WANT TO UNDERSTAND?
How much do recommended products and displayed sources change when we repeat a question or use a different access method?
The researchers examined which products assistants recommend, how recommendations are presented and which sources accompany them. They also investigated whether external auditors can use API responses as a substitute for the consumer interface. That distinction matters when establishing which system a visibility report actually describes.
Explaining the context · DijitalPi commentary
DijitalPi commentary: The usefulness of a product mention depends partly on how consistently it can be observed. A marketing team allocating content budget from a single screenshot may overlook how the result depends on the question, timing and interface. Using this paper to assess measurement practices is a more cautious inference than treating source lists as direct instructions for investment.
02 / HOW WAS THE RESEARCH CONDUCTED?
How did they test the question?
The researchers selected 117 physical-product questions from ConsumerQ, a pool of 2,528 real commercial-advice questions verified through human annotation from the open conversation datasets LMSYS-Chat-1M and WildChat-1M. Each question was submitted 3 times under 5 conditions: the ChatGPT interface, ChatGPT API, Gemini interface, Gemini API and Google AI Overviews. This observational audit used residential IP addresses in the Netherlands. 1,536 of 1,755 observations produced responses. AI Overviews searches took place on 4–5 September 2026. Chatbots were examined within the same collection window, but separate dates were not specified.
Understanding the method · DijitalPi commentary
Highly specific questions and those containing geographical information were excluded. The middle value when question lengths were ranked, called the median, was 8 words. The researchers kept the original spelling. Each interface observation used a fresh browser profile, a logged-out session and no conversation history. Comparisons covered repetitions of the same question, different systems and matched interface–API responses. There was no randomly assigned control group. When the gaps between paired interface and API requests were ranked, the middle value was 0.17 seconds. The logged-out ChatGPT interface’s model could not be independently verified, so it was not assumed to match the API model.
The researchers assessed recommendation language, recommended products and displayed sources. Language labels were produced with a language model validated against human annotations. Consistency was assessed using Jaccard overlap: the number of shared items divided by the total number of distinct items across two lists. Source analysis covered displayed citations only. Their contribution to the answer was not measured. Only 134 of 351 AI Overviews searches (38.2%) yielded recoverable content, and content and source analyses were restricted to those observations.
03 / RESEARCH FINDINGS
Repeated questions and different access methods did not yield the same product and source lists.
The researchers found mean product overlap across repetitions of 0.178 for ChatGPT, 0.287 for Gemini and 0.421 for AI Overviews. Different spellings of product names were counted separately, making these figures lower bounds of actual overlap. For the same question, displayed domains in the ChatGPT and Gemini interfaces had a mean overlap of 5.4%. 76.7% of comparisons shared no domain. Within-system domain overlap across repetitions was 26.0% for ChatGPT, 29.8% for Gemini and 45.9% for AI Overviews. Matched interface–API comparisons showed mean domain overlap of 12.0% for ChatGPT and 14.8% for Gemini. APIs displayed sources more frequently than their interfaces, by 9.7 and 8.1 percentage points respectively. These are observed differences between audit conditions, not effects attributable solely to the access method.
ChatGPT product overlap across repetitions
0.178Mean overlap between recommended product sets across repetitions of the same question. Variation in product-name spelling makes this a lower bound of actual overlap. It is neither an appearance probability nor a sales rate.
ChatGPT interface–API domain overlap
12.0%Mean overlap of displayed source domains in responses matched by question and repetition. It does not establish how much those sources influenced the recommendation.
DijitalPi commentary: A visibility report should show the variability of its observations alongside the answers that mention your product. Repeating unchanged questions and separately comparing API results with the consumer interface are possible ways to investigate that variability. Treat displayed sources as candidates for further examination. Their appearance does not establish that being featured on those websites will secure a recommendation.
The underlying conversation datasets do not represent the general population and lean towards users interested in technology. The audit covered only 117 general physical-product questions and 3 repetitions, while the systems continue to change. The cause of interface–API divergence could not be established. Additional source-type classifications were not systematically validated by humans and remain exploratory, providing insufficient grounds on their own for allocating budget to a particular source category.
05 / CONCLUSION AND OPEN QUESTIONS
What did we learn, and what do we still not know?
This preprint indicates that repetition and interface choice matter when assessing visibility in product recommendations. Its findings concern predominantly English questions answered from the Netherlands within a particular collection window; they do not represent Dutch consumer behaviour and have not been validated for Turkish or Türkiye. Clicks, sales, advertising effects and sources’ contributions to answers were not measured. The authors are university researchers. No funding statement or formal competing-interest declaration was found in the source. The paper states that Samsung did not sponsor it, referring to the product example in its title. The authors disclose using large language models for coding, writing and annotation work.
Our open question at DijitalPi: how much does your visibility report change when you repeat Turkish product questions under consistent conditions, and which budget decision would that variability lead you to reconsider?
We created these scenarios to make the topic concrete. They are not cases from the paper or measured client results.
EXAMPLE 01
E-commerce: a product is missing from one answer
An illustrative camping equipment shop cannot find its product in an answer to a general tent recommendation question.
Is this answer enough to conclude that our product is invisible in AI?
Open the recommendation for example 01+
Compare repeated answers before deciding to change your content.
Keep the question unchanged, collect consumer-interface answers at different times and record the conditions. Use separate columns for whether the product appears, its stated name and the sources displayed in each answer. The paper’s repetitions do not capture every possible answer. Your observations will likewise not establish complete visibility or sales effects.
EXAMPLE 02
Retail: a displayed source becomes a budget decision
An illustrative stationery retailer is considering allocating content budget to a review website cited in a product recommendation.
Does a citation mean that appearing on that website will get us recommended?
Open the recommendation for example 02+
Record the citation, then examine the separate question of its contribution to the recommendation.
Repeat the same question across assistants and compare the displayed domains. Record product mentions separately from citations of the review website. Do not report their co-occurrence as a causal result. The paper did not measure sources’ contributions to answers, so this observation alone cannot establish the return on content investment.
EXAMPLE 03
B2B equipment: an API report differs from the interface
An illustrative office equipment supplier tracks its visibility in printer recommendations through an API-based report.
Does this report represent the answer a purchasing team sees on screen?
Open the recommendation for example 03+
Compare API results with consumer-interface answers to the same questions.
Submit your selected questions to the API and consumer interface close together in time and save both outputs. Compare recommended products and cited domains separately, and keep interface and API results distinct in the report. Do not attribute divergence solely to the access method; this study does not validate B2B purchasing behaviour or outcomes in Türkiye.
TRY IT WITH YOUR TEAM
Repeat product questions and compare the answers.
Select the product questions informing your decisions and plan repeated consumer-interface observations using unchanged wording. If you use an API report, compare its answers to the same questions, assessing product mentions, displayed sources and commercial outcomes separately. This is a measurement proposal that DijitalPi has not implemented.
Review your content to make product information clear and supported by sources. SEOMACHINE brings brand information into content preparation and supports source checks. Whether AI recommends your product is a separate outcome to measure. Explore SEOMACHINE ↗
Lucas G. Uberti-Bona Marin, Thales Bertaglia, Giovanni Astante, Bram Rijsbosch, Gijs van Dijck, Anikó Hannák, Gerasimos Spanakis, Konrad Kollnig Law & Tech Lab, Maastricht University; Utrecht University; Social Computing Group, Department of Informatics, University of Zurich.
The academic work belongs to the researchers named above. This page contains DijitalPi's explanatory review and original business examples; it is not a full translation of the paper.
Cookie preferences