Does changing the AI model change which product it chooses?
Do different AI models pick the same product in a test shop when prices and customer ratings stay unchanged? GPT-4.1 selected the same phone case in 62.6% of trials, while the newer GPT-5.1 selected it in 5%. When AI shopping on a customer’s behalf is updated, a product’s chance of being selected can change even if the seller changes nothing.
ACADEMIC WORK REVIEWED
What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commerce
English review: 18 October 2026 · The source date appears in the citation above.
Editorial review draft
The source has been checked; the review still carries its original editorial status. DijitalPi did not conduct a customer experiment for this paper.
LET'S READ THE RESEARCH TOGETHER
We first explain the researchers' question, method and findings. We then discuss how to interpret the results, clearly separating DijitalPi's commentary from the source.
01 / WHAT DID THE RESEARCHERS WANT TO UNDERSTAND?
Which products does AI choose when shopping for us, and do those choices depend on the model?
The researchers tested whether AI models shopping for customers followed basic instructions. They examined how product information and listing layouts affected choices. They also tested whether model updates or a seller’s description changes altered how often products were selected.
Explaining the context · DijitalPi commentary
DijitalPi commentary: AI preferences can change even when prices and descriptions stay the same. A choice cannot therefore be attributed to product quality or wording alone. Businesses should record the model and listing layout used for each comparison.
02 / HOW WAS THE RESEARCH CONDUCTED?
How did they test the question?
Allouah, Besbes, Figueroa, Kanoria and Kumar built ACES, a controlled shopping simulator. It contained 8 product categories with 8 products each, drawn from publicly available Amazon information. The study covers the August 2025 models Claude Sonnet 4, GPT-4.1 and Gemini 2.5 Flash, alongside their successors as of 1 December 2025: Claude Opus 4.5, GPT-5.1 and Gemini 3.0 Pro Preview. Initial experiments ran in early August 2025, with repeats examining changes over time in late September 2025. Basic instruction tests used 50 trials per task-category, selection-share analysis used 1,000 experiments per category, and choice-behaviour analysis used 500 random scenarios per category. The study did not examine real purchases by human participants.
Understanding the method · DijitalPi commentary
The AI read a screenshot of the mock shop’s product grid, arranged in 2 rows and 4 columns, and made a single-step choice. There were no product-detail pages. Shopping instructions were generic and unpersonalised, and reasoning was set to the lowest viable level. The researchers randomly assigned positions and labels and randomly varied prices, ratings and review counts. This allowed comparisons such as displays carrying a sponsored label versus those without one.
The main outcome was the proportion of trials in which a product was selected. What the authors call market share represents a hypothetical setting where shopping is fully delegated to AI. It is not the share of products sold in a real market. In the description experiment, a seller AI rewrote one product’s text once, followed by 200 trials before and 200 after the change, using the same product positions. The comparison was between revised and original descriptions. It did not test competition in which other sellers revised their descriptions simultaneously.
03 / RESEARCH FINDINGS
Different models preferred different products in the same list
The researchers report that GPT-4.1 selected Mikeke in 62.6% of iPhone 16 Pro case trials, whereas GPT-5.1 selected ESR in 95% and Mikeke in 5%. Labels affected choices even when a product’s position stayed the same. For a product with a starting selection probability of 10%, GPT-4.1’s probability fell to 8.0% with a sponsored label and rose to 19.9% with a platform recommendation label. The scarcity label slightly reduced selection probability, or the measured difference was too uncertain to establish an effect. A single description change produced a statistically significant increase in selection rates in 33% of experiments. The average change was +14.89 percentage points for GPT-5.1. The +0.32 percentage-point estimate for Gemini 3 Pro Preview was too uncertain to establish an increase. Some products were selected less often after their descriptions changed. These results do not measure actual sales or revenue.
Mikeke selection rate with GPT-4.1
62.6%The product’s share of selections in the iPhone 16 Pro case simulation, not its share of actual market sales.
Mikeke selection rate with GPT-5.1
5%Its selection share with the successor model in the same category, showing that the earlier preference did not persist.
DijitalPi commentary: One successful test is not enough to set a rule for every product description. Repeat comparisons and record the model version and product positions. One test could check whether accurate information about the product’s intended use appears in the visible description. Assess the effect of a sponsored label separately from the effect of placing an advert near the top of the list.
Office lamps were the only category where description changes produced consistent, statistically significant increases across all six buying models. The authors suggest that the word office appeared late in the original description and may have fallen outside the opening portion shown in the list. They have not confirmed this explanation. The revised descriptions were not meaningfully shorter. The findings do not support shortening every description, expecting sales gains from adding keywords or dropping sponsored advertising.
05 / CONCLUSION AND OPEN QUESTIONS
What did we learn, and what do we still not know?
This study is a preprint, meaning it has not yet undergone peer review. It shows that product choices can vary by model version, listing layout and labels. It does not explain why these preferences arise. The AI made a single choice in a mock shop using instructions that were not personalised. The study did not measure actual sales, revenue or customer satisfaction. It did not cover the full shopping process or directly compare AI choices with human shoppers. Trials with technical problems were discarded. Products that were never selected were left out of the statistical model’s calculations. The results concern the versions tested; their validity for Turkish, Türkiye or every industry has not been established. Allouah and Figueroa are affiliated with MyCustomAI, whose GitHub account hosts the ACES code. No funding or competing-interest declaration was found in the source. No relationship with the model providers studied was stated.
Our open question at DijitalPi: which preferences persist in your Turkish product listings when the model version changes, and does revising the visible description produce a consistent change in those preferences?
We created these scenarios to make the topic concrete. They are not cases from the paper or measured client results.
EXAMPLE 01
E-commerce: product selection after a model update
An illustrative phone accessories retailer wants to understand how different AI versions choose from the same product list.
Is my product being selected less because of its description or a model change?
Open the recommendation for example 01+
Separate the model comparison from the description comparison.
Keep prices, ratings, review counts, descriptions and layout unchanged while recording selections from the model versions you can access. Then change only the description and examine selection rates separately for each version. If the older version is unavailable, do not attribute historical differences solely to the update. These selections are not completed sales.
EXAMPLE 02
B2B supplies: a use case hidden from view
An illustrative office equipment supplier’s lamp description places its intended use in text that gets cut off in the listing.
Does the visible text make the product’s suitability for office use clear?
Open the recommendation for example 02+
Move accurate information about its use into the visible text.
Inspect the text shown in the product list and bring a genuinely supported use case to the beginning. Compare the original and revised descriptions using the same product positions and model version, recording selection rates. The researchers’ explanation for the office lamp finding is a hypothesis. This scenario does not establish an increase in business purchases or quotation requests.
EXAMPLE 03
Retail: separating the sponsored label from placement
An illustrative household goods retailer is considering reducing its advertising budget after reading that AI may select sponsored products less often.
Does the response to a sponsored label capture the advertisement’s total value?
Open the recommendation for example 03+
Assess the label separately from the visible position advertising provides.
In a controlled trial, first hold position constant and compare displays with and without a sponsored label. Then examine actual campaign placement and completed orders in separate records. The paper measured label effects in a mock shop. It does not show what reducing an advertising budget would achieve.
TRY IT WITH YOUR TEAM
Record the comparison conditions
Choose a product category and plan a trial comparing identical information across model versions, followed by a comparison that changes only the description. Assess selection rates separately from actual orders; this is a DijitalPi proposal that has not yet been implemented.
The academic work belongs to the researchers named above. This page contains DijitalPi's explanatory review and original business examples; it is not a full translation of the paper.
Cookie preferences