Did AI-written subject lines lead to more purchases in Mercari’s email trial in Japan? Product click rates rose by a relative 23.63% compared with a fixed subject line, but there was no significant increase in purchase rates. More clicks from an email subject may not translate into more sales.
ACADEMIC WORK REVIEWED
Using item recommendations and LLMs in marketing email titles
ResearchersDeddy Jobson, Muktti Shukla, Phuong Dinh, Julio Christian Young, Nick Pittoni, Nina Chen, Ryan Ginstrom
arXiv · 21 September 2025 · v3; accepted to the GenAIECommerce ’25 workshop · Preprint · Source language: English
English review: 25 October 2026 · The source date appears in the citation above.
Editorial review draft
The source has been checked; the review still carries its original editorial status. DijitalPi did not conduct a customer experiment for this paper.
LET'S READ THE RESEARCH TOGETHER
We first explain the researchers' question, method and findings. We then discuss how to interpret the results, clearly separating DijitalPi's commentary from the source.
01 / WHAT DID THE RESEARCHERS WANT TO UNDERSTAND?
Can AI subjects describing recommended products improve email engagement and cover their production costs?
The researchers examined replacing a fixed email subject template with subjects connecting recommended products through a theme. They asked whether open rates would improve and whether the value linked to subsequent purchases could offset AI usage costs. The paper also describes the quality checks used to prepare these subjects for deployment.
Explaining the context · DijitalPi commentary
DijitalPi commentary: Changing a reactivation email’s subject may seem like a small production decision. Yet opens, product clicks and purchases answer different questions. The marketing value of this study lies in showing that a subject change with unchanged content did not produce equally clear improvements across those outcomes.
02 / HOW WAS THE RESEARCH CONDUCTED?
How did they test the question?
Researchers employed by Mercari randomly assigned more than a million users of its Japanese second-hand marketplace to two equally sized groups. Eligible users had not opened the app in the last 7 days, had opened it at least once in the last 1 year and had made at least one purchase in the last 6 months. The experiment lasted one week, with one Japanese email sent to each user per week. An assignment method based on user identifiers kept users in the same group.
Understanding the method · DijitalPi commentary
The control subject used a fixed template naming the first recommended product and stating that it and other products were available. For the treatment group, GPT-4o-mini generated a thematic subject from the products recommended in the email. Content came from the same recommendation algorithm. The subject was the element changed. User profiles were not separately supplied for subject generation. The research question prioritised open rates, while product clicks and buyer conversion were also tracked. Buyer conversion was defined as buyers divided by target users.
Before deployment, 1,000 sample subjects underwent manual review. Reviewers found repetition, excessive length, unnatural Japanese and sensitive products, among other issues. The process included a sensitive-word filter, preliminary suitability checks using other language models and reviews by relevant company teams. A note at the end of the emails disclosed that the subject was AI-generated. The experiment therefore did not evaluate sending unchecked subjects directly to recipients.
03 / RESEARCH FINDINGS
Product clicks increased, without significant gains in opens or buyer conversion
Table 2 reported a statistically significant 23.63% relative increase in the email item click rate compared with the fixed-subject group. The relative changes of +0.46% in open rate, 45.12% in buyer conversion via email and −0.42% in overall buyer conversion were not statistically significant. The reported email buyer conversion result therefore cannot be presented as a sales gain. Send rate showed a statistically significant relative change of −0.90%, which the authors did not explain. Absolute rates and confidence intervals were not provided, and the denominator of the product click rate was undefined. These are not percentage-point differences. Weekly API costs were estimated at around USD 300, but the paper provided no numerical answer on whether those costs were recovered.
Email item click rate
+23.63%Relative increase compared with the fixed-subject group. Statistically significant. The paper does not define the rate’s denominator.
Overall buyer conversion rate
−0.42%Relative change in the proportion of target users who became buyers. Not statistically significant.
DijitalPi commentary: If you test subject generation, treat product clicks as an intermediate outcome and purchasing as a separate decision measure. Track how many emails are sent to each group, given the unexplained sending decline in this study. Record the count you divide by when calculating each rate. Keep starting and final values as well as the change. These records would supply information missing from the paper for your own test.
The authors suggest that thematic subjects better explained what recipients would find inside, but this mechanism was not separately tested. Failure to detect a significant purchasing increase does not establish that there was no effect: the researchers describe the results as inconclusive and want to repeat the experiment. Japanese wording, one platform, a selected reactivation audience and a one-week duration constrain interpretation; Türkiye and Turkish-language campaigns require separate assessment.
05 / CONCLUSION AND OPEN QUESTIONS
What did we learn, and what do we still not know?
These results do not provide a forecast of gains for Turkish-language campaigns. This randomised experiment supports an increase in product clicks under the conditions studied. It does not offer the same support for opens, buyer conversion or cost recovery. The reviewed v3 text is an arXiv preprint accepted as a workshop paper, rather than a full journal article. Findings should be assessed within the scope of Mercari users in Japan and Japanese subjects. All seven authors work for Mercari, which supported the research and provided the experimental platform. No separate funding or competing-interest statement was found.
Our open question at DijitalPi: if product clicks rise in your Turkish campaign while the purchasing outcome remains uncertain, which measure will guide whether you continue generating subjects?
We created these scenarios to make the topic concrete. They are not cases from the paper or measured client results.
EXAMPLE 01
E-commerce: a shared theme for recommended products
An illustrative clothing retailer wants to email selected products to customers who have not purchased recently.
Can a subject connecting the products contribute to purchasing?
Open the recommendation for example 01+
Keep the product list fixed and compare subjects in randomly assigned groups.
Compare the existing template with a thematic subject generated solely from the products in the email. Record emails sent, users clicking products and users purchasing separately, specifying each rate’s denominator in advance. Have a person check that subjects promise no features absent from the products. Evidence from a Japanese second-hand marketplace does not establish the same outcome for a Turkish clothing campaign.
EXAMPLE 02
Tourism: a subject describing accommodation options
An illustrative accommodation business is preparing an email of suitable room options to bring previous guests back.
Are more room-link clicks enough to call the subject successful?
Open the recommendation for example 02+
Track link clicks and completed bookings as separate success measures.
Keep email content and sending time consistent when comparing the existing subject with one summarising the options. Check the subject against room features and booking conditions. Report clicks and completed bookings separately over a predefined observation period. The study did not examine accommodation bookings, so this scenario is a new application proposal.
EXAMPLE 03
B2B supply: clicks without a clear order outcome
An illustrative business equipment supplier plans to email relevant products to previous customers.
How can I assess whether AI subjects cover their costs?
Open the recommendation for example 03+
Consider generation costs, review work and completed orders together.
Randomly assign recipients to emails showing the same products with either the existing or AI-generated subject. Record delivered emails, product clicks, quotation requests and orders separately. Include the costs of subject generation and human review in the assessment. The paper gives no numerical answer on cost recovery, so do not treat a click increase as an economic gain.
TRY IT WITH YOUR TEAM
Define how your subject test will measure purchases
For a subject comparison with identical product content, specify the purchasing measure, observation period and how you will investigate sending differences. Review Turkish subjects for meaning, brand tone and sensitive wording. This is an illustrative experiment proposed by DijitalPi that has not been conducted.
When comparing subject lines, you can keep the email body the same. MAILMACHINE turns your prepared content into HTML and plain-text email packages and checks their presentation. Your team prepares the subject lines; your existing email system sends the messages. Explore MAILMACHINE ↗
The academic work belongs to the researchers named above. This page contains DijitalPi's explanatory review and original business examples; it is not a full translation of the paper.
Cookie preferences