Did faster AI support leave customers happier in a trial at Taobao in China? Chats suitable for AI became shorter, but customer ratings fell by 0.412 points on a 1-5 scale compared with the human-support group. Faster service may not mean happier customers.
ACADEMIC WORK REVIEWED
Agentic AI and Human-in-the-Loop Interventions: Field Experimental Evidence from Alibaba’s Customer Service Operations
English review: 22 October 2026 · The source date appears in the citation above.
Editorial review draft
The source has been checked; the review still carries its original editorial status. DijitalPi did not conduct a customer experiment for this paper.
LET'S READ THE RESEARCH TOGETHER
We first explain the researchers' question, method and findings. We then discuss how to interpret the results, clearly separating DijitalPi's commentary from the source.
01 / WHAT DID THE RESEARCHERS WANT TO UNDERSTAND?
How does AI that replies to customers under human supervision affect service speed and quality?
The researchers examined an AI system that could conduct customer conversations under human supervision. They also investigated the circumstances in which people took over and how service changed after handover. This brought automated handling and human intervention into the same assessment.
Explaining the context · DijitalPi commentary
DijitalPi commentary: A shorter support conversation provides limited information about the customer’s experience on its own. This study matters for decisions because speed and customer ratings could move in opposite directions, while automation could also affect other conversations handled by people. A business assessment should therefore include which questions are automated and when a person takes over.
02 / HOW WAS THE RESEARCH CONDUCTED?
How did they test the question?
Alibaba conducted a field experiment in Taobao customer service in China during August 2024, covering 647 randomly selected workers and 680,676 chats. Independent remote workers paid on a piece-rate basis were randomly assigned by employee ID to a control group of 345 and a treatment group of 302. Everyone handled chats themselves during 1-14 August. During 15-31 August, supervised AI answered eligible chats for the treatment group. The main sample contained 39,432 AI-eligible chats, representing 5.8% of the sample.
Understanding the method · DijitalPi commentary
Control workers continued to answer every chat themselves. In the treatment group, AI handled eligible routine chats while workers monitored it, intervened when needed and continued to handle ineligible chats. The researchers compared changes from before to after deployment between the groups. Speed meant elapsed time from chat initiation to termination. Quality was assessed through customer ratings on a 1-5 scale and repeat contact about the same issue within 7 days. Ratings were available for 115,243 chats, or 17% of chats.
Handovers occurred when the system detected a technical limitation or negative sentiment, or when a worker chose to intervene. To examine handover types, the researchers matched 11,069 chats handled by supervised AI one-to-one with similar human-handled chats in the control group. Handover type was not randomly assigned in this analysis, and matching could balance only observed characteristics. The overall experiment therefore provides stronger evidence of cause and effect. Results for individual handover types do not provide equally strong evidence.
03 / RESEARCH FINDINGS
AI-eligible chats became faster, but customer ratings fell
Across all chats, the researchers found that average duration fell by 3.2%. Overall effects on repeat contact and customer ratings were not statistically significant. In AI-eligible chats, duration fell by 16.8% and customer ratings fell by 0.412 points on the 1-5 scale relative to the control group. The repeat-contact difference was not significant in the main analysis, although an additional worker-hour analysis found a significant increase. In AI-ineligible chats handled by people, duration fell by 1.8% and customer ratings rose by 0.091 points.
In the matched analysis, technical handovers were associated with 19.1% longer chats than comparable conversations handled entirely by people, with no significant differences in ratings or repeat contact. Automatic handovers prompted by negative sentiment were associated with 40.8% longer chats, a repeat-contact rate 6 percentage points higher and ratings 0.928 points lower. Worker-initiated handovers were associated with 9.5% longer chats and ratings 0.524 points lower. The 3.2 percentage point reduction in repeat contact had weaker statistical support (p < 0.1). Chats completed by AI without a handover were 64.6% shorter and received ratings 0.858 points lower, with no significant repeat-contact difference. These handover comparisons show associations. They do not establish that a particular handover rule caused the outcomes.
Duration of AI-eligible chats
16.8%Reduction in chat duration in the main experimental analysis. This is not a measure of sales or revenue growth.
Customer ratings for AI-eligible chats
0.412 pointsDecline relative to the control group on the 1-5 scale. This is neither a percentage nor a percentage point change.
Customer ratings for human-handled chats not suitable for AI
0.091 pointsIncrease in customer ratings for these chats, assessed separately from chats suitable for AI.
DijitalPi commentary: Report pilot results separately for AI-eligible and ineligible conversations. Alongside faster handling, make lower customer ratings, repeat contact and handover timing visible. Rules that allow intervention before frustration builds are worth testing, but the paper does not provide a ready-made intervention threshold for your business.
The authors link improvements in other human-handled chats to less switching between conversations, and lower ratings in fully AI-handled chats to communication style. These explanations have not been confirmed for every business. The findings on handover timing describe what happened, and worker-initiated handovers were not classified as technical or emotional. The significant increase in repeat contact in the additional analysis also means that the non-significant main result should not be read as assurance that resolution quality was preserved under all conditions.
05 / CONCLUSION AND OPEN QUESTIONS
What did we learn, and what do we still not know?
This study is a preprint, meaning it has not yet undergone peer review. Sales and revenue were not measured. Customer ratings were available for only 17% of chats. This preprint shows that faster handling and lower customer ratings can occur together in certain routine conversations, while the overall effect on ratings was not significant. The findings concern Taobao in China during August 2024, remote workers paid on a piece-rate basis and a limited set of eligible chats; they do not establish applicability to Turkish-language services or businesses in Türkiye. Potential long-term costs of specialist supervision, including emotional depletion and skill erosion, were not measured. The study was conducted in partnership with Alibaba, which ran the experiment on its own system. Author Bingxin Jia is an Alibaba employee. Text-based effort and sentiment analyses used Alibaba’s Qwen3.5-122B-A10B model on the company’s intranet. No funding statement or separate competing-interest declaration was found in the source.
Our open question at DijitalPi: if your Turkish-language support chats become shorter but customer ratings fall, which automation or handover decision will you change?
We created these scenarios to make the topic concrete. They are not cases from the paper or measured client results.
EXAMPLE 01
E-commerce: delivery chats end quickly
An illustrative e-commerce business is considering handing routine delivery questions to AI.
Could shorter chats hide unresolved customer problems?
Open the recommendation for example 01+
Track customer ratings and repeat contact alongside speed.
Plan a pilot that allows comparison with human responses to similar delivery questions. Record start and end times, customer ratings on a 1-5 scale and repeat contact about the same issue within 7 days. Show the share of customers submitting ratings. Do not count unrated chats as satisfaction. The paper’s reduction in duration is not an expected gain for this business.
EXAMPLE 02
Local services: a booking change becomes tense
An illustrative maintenance business is considering AI live chat for appointment changes.
When should a person take over if the customer objects?
Open the recommendation for example 02+
Define possible signals for earlier intervention and test them in a pilot.
Agree on signals to investigate, such as a customer repeating the same objection. Record the timing and reason for each handover and the subsequent customer rating. Where feasible, compare handover rules using random assignment. The paper’s timing analysis is descriptive. It does not establish that earlier intervention will improve outcomes for this business.
EXAMPLE 03
B2B software: routine questions and specialist support
An illustrative business software provider plans to assign routine usage questions to AI and complex issues to its support team.
Could automation also affect complex chats handled by people?
Open the recommendation for example 03+
Track duration and customer ratings for human support separately.
Define routine and specialist question categories before the pilot starts. Report human-handled chats separately for the team receiving AI support and the comparison team. Alongside duration and ratings, examine how often workers switch between chats. The researchers’ attention-based explanation does not guarantee the same effect here or establish the long-term benefits of a specialist supervisor role.
TRY IT WITH YOUR TEAM
Measure time and customer satisfaction together
Select a routine question category and design a pilot that can be compared with human support. Prepare a recording plan covering duration, customer ratings, rating response rates, repeat contact and handover timing. This is a proposal that DijitalPi has not implemented.
PiAIMode can answer questions about products and services on your site. It bases answers on your site content, and visitors can contact you through an in-chat form or WhatsApp. Handling orders and refunds requires a separate integration. Explore PiAIMode ↗
Yiwei Wang; Chuan Zhu; Tianjun Feng; Lauren Xiaoyuan Lu; Bingxin Jia. Zhejiang University, International Business School (Wang); Fudan University, School of Management (Zhu, Feng); Dartmouth College, Tuck School of Business (Lu); Alibaba Group Inc. (Jia).
The academic work belongs to the researchers named above. This page contains DijitalPi's explanatory review and original business examples; it is not a full translation of the paper.
Cookie preferences