A customer describes what they need in your site’s search box and receives a product list. Your team considers adding an AI answer above the results. Before committing development budget, you want to know: could this help customers complete an order? In the reviewed SSRN author manuscript of this peer-reviewed article, researchers found that access to this feature increased order numbers by 0.174% in a randomised field experiment on Meituan in China. DijitalPi commentary: Assessing such a feature calls for tracking completed orders alongside clicks. The reported increase is small and relative; the analysis includes users who never saw an answer. The study does not measure your business’s profit, organic traffic to external websites or outcomes in Turkish.
ACADEMIC WORK REVIEWED
Generative Search: Evidence from a Large-Scale Field Experiment
ResearchersShuang Zheng, Yuting Zhu, Xin Ye, Liang Shen.
Management Science / SSRN · 13 May 2026 · Reviewed SSRN author manuscript; published online in the journal 24 September 2026 · Peer-reviewed article · Source language: English
English review: 10 October 2026 · The source date appears in the citation above.
Editorial review draft
The source has been checked; the review still carries its original editorial status. DijitalPi did not conduct a customer experiment for this paper.
LET'S READ THE RESEARCH TOGETHER
We first explain the researchers' question, method and findings. We then discuss how to interpret the results, clearly separating DijitalPi's commentary from the source.
01 / WHAT DID THE RESEARCHERS WANT TO UNDERSTAND?
How does an AI answer above search results change purchasing and the way people search?
The researchers asked whether offering AI answers to consumers who describe their needs in natural language increases orders. They also examined changes in queries, merchant browsing and clicks. This allowed them to assess whether more search activity is necessary for purchasing.
Explaining the context · DijitalPi commentary
DijitalPi commentary: When adding a search feature, it is easy to treat time spent or click numbers as the measure of success. Yet a customer who understands their requirements sooner may need fewer pages. This research suggests why browsing indicators and commercial outcomes can usefully appear in the same assessment.
02 / HOW WAS THE RESEARCH CONDUCTED?
How did they test the question?
A randomised field experiment ran on the main app of Meituan, a Chinese local products and services platform, from 5 September to 5 October 2024. It included 6,451,767 users who had previously submitted at least one query describing their underlying needs in natural language. The introduction reports that 3,227,126 were assigned to generative search and 3,224,641 to keyword-based search. Search behaviour data were available only for a randomly selected subsample of 90,000 users: 45,000 from each group.
Understanding the method · DijitalPi commentary
Generative search added an AI-written answer above the results list. The treatment group received answers only for queries expressing underlying needs; both groups saw normal rankings for straightforward keyword queries. The control group always received the existing results list, regardless of query type. Meituan’s LongCat model generated the answers, while other pages and the underlying search and recommendation algorithms remained unchanged. Order rate meant each user’s total orders divided by total search queries; gross transaction value meant total spending during the experiment, rather than platform revenue or profit.
The main analysis included everyone assigned to the treatment group, whether or not they saw an answer. In the subsample, 39,305 of its 45,000 treatment users never submitted a query that would trigger an answer during the experiment. The headline result therefore cannot be read as the average effect on actual answer viewers. Sessions originating from the related searches section accounted for approximately 2% of sessions and were excluded. The Table 2 note gives the control group size as 3,224,616; this review uses the figure from the introduction. Absolute outcome levels were withheld for confidentiality.
03 / RESEARCH FINDINGS
Orders increased slightly while total time on the platform fell.
The researchers found increases of 0.174% in total orders per consumer, 0.072% in order rate and 0.261% in gross transaction value compared with the control group. All three were statistically significant, with respective p-values of 0.002, 0.008 and 0.005. These are percentage changes relative to the control group mean, not percentage points. In the behavioural subsample, total time on the platform fell by 3.64%, while session numbers showed no significant change. The number of merchants browsed fell by 1.87%, although this result met only the weaker p < 0.1 threshold. The number of distinct merchants clicked fell by 2.25%; the decrease in total clicks was not significant. Click-through rate, defined here as total clicks divided by the number of merchants browsed, increased by 0.61 percentage points. The change in unique click-through rate was small and statistically insignificant.
Total orders per consumer
0.174%Increase relative to the control group mean, covering assigned treatment users including those who never saw an answer.
Gross transaction value per consumer
0.261%Relative increase in total spending during the experiment, not a measure of business profit or platform revenue.
DijitalPi commentary: Because less browsing can accompany more orders, making clicks the sole target could be misleading. Our proposed assessment of an answer layer would track orders and spending alongside search behaviour. Clearly stating product or service attributes is also a content recommendation we draw from the authors’ explanation; that particular change was not separately tested.
The authors judge the behavioural patterns most consistent with answers helping users understand their needs and relevant attributes. However, they did not observe answer content, which prevents them from directly separating the mechanisms. The sample was more highly educated than the platform’s overall user base and, according to the authors, likely to have greater previous exposure to AI. The effect sizes cannot be transferred directly to other user groups, Türkiye, Turkish-language use or different sectors.
05 / CONCLUSION AND OPEN QUESTIONS
What did we learn, and what do we still not know?
This experiment shows that adding AI answers to search can produce small increases in orders and spending within a selected user group. Effects on sponsored search advertising could not be investigated in depth, and the presence of related searches may also have influenced how answers were perceived. The reviewed text is the SSRN author manuscript dated 13 May 2026. The study was published online in Management Science on 24 September 2026; the journal version was not read, so identical figures and wording cannot be assumed. Meituan provided the experimental setting and data, the study involved its search algorithm team, and author Liang Shen was a company employee. No funding statement or separate conflict-of-interest declaration was found in the reviewed source.
Our open question at DijitalPi: in your Turkish-language site search, do users given access to an answer layer complete more orders while browsing less, or does only their search behaviour change?
We created these scenarios to make the topic concrete. They are not cases from the paper or measured client results.
EXAMPLE 01
E-commerce: turning needs into product attributes
An illustrative luggage retailer’s customer wants a bag suitable for cabin travel that can also carry a laptop.
Could an answer explaining those needs support product selection and ordering?
Open the recommendation for example 01+
Keep product rankings and other conditions unchanged when testing the answer layer.
Plan to randomly assign eligible users to two groups, showing answers to needs-based queries in one group and the existing results list in the other. Compare order numbers, orders per query and total spending across everyone assigned to each group. Count actual answer viewers separately. The effect size from a Chinese local services platform is not an expected gain for this retailer.
EXAMPLE 02
Tourism: choosing accommodation with less browsing
An illustrative accommodation platform’s customer searches for somewhere quiet to work remotely.
Does a shorter visit mean the customer has given up?
Open the recommendation for example 02+
Assess time spent, properties viewed and completed bookings together.
Record total time and the number of distinct properties clicked when comparing existing search with an answer layer. Track completed bookings separately; do not classify reduced time alone as success or failure. Include a review of which property attributes the answers emphasise. The paper did not observe answer content, so it cannot establish that particular wording will increase bookings.
EXAMPLE 03
B2B: from a needs-based search to a quotation request
An illustrative warehouse equipment supplier’s customer describes a need for shelving suitable for a confined space.
Can apparently more relevant clicks count as commercial results?
Open the recommendation for example 03+
Measure product exploration, quotation requests and completed orders as separate stages.
Track search queries, product clicks and quotation requests within the same assessment. Record any resulting order separately from the request. Because the Meituan study’s main outcome concerned actual orders, do not present a quotation request as an equivalent result. This B2B scenario was not tested in the research.
TRY IT WITH YOUR TEAM
Set up order measurement first.
Specify which needs-based queries will trigger the answer layer and how you will compare it with existing search. Plan to link order, query and spending records to group assignment while also recording whether an answer was actually displayed. This is a proposed pilot that DijitalPi has not carried out.
Shuang Zheng, Yuting Zhu, Xin Ye, Liang Shen. Shuang Zheng — Renmin University of China; Yuting Zhu — National University of Singapore, NUS Business School; Xin Ye — Dalian University of Technology; Liang Shen — Meituan.
The academic work belongs to the researchers named above. This page contains DijitalPi's explanatory review and original business examples; it is not a full translation of the paper.
Cookie preferences