If one ad brings customers at a lower cost, is that because of its message or its audience? An A/B test compares alternative ads. A Lift test compares people who can see an ad with people who cannot. In this preprint, the researchers report frequent differences between audiences reached in Meta A/B tests, while detecting no imbalance in Lift tests with a no-ad control group. Peer review is not documented for this version. DijitalPi commentary: Examine who each ad reached before drawing a general lesson about its message. The main outcome is audience balance, not sales growth. The main sample’s country coverage is unspecified; findings cannot be assumed to apply to Türkiye or Turkish-language advertising.
ACADEMIC WORK REVIEWED
Characterizing and Minimizing Divergent Delivery in Meta Advertising Experiments
ResearchersGordon Burtch, Robert Moakler, Brett R. Gordon, Poppy Zhang, Shawndra Hill
arXiv · 28 August 2025 · v1, peer review not documented · Preprint · Source language: English
English review: 13 October 2026 · The source date appears in the citation above.
Editorial review draft
The source has been checked; the review still carries its original editorial status. DijitalPi did not conduct a customer experiment for this paper.
LET'S READ THE RESEARCH TOGETHER
We first explain the researchers' question, method and findings. We then discuss how to interpret the results, clearly separating DijitalPi's commentary from the source.
01 / WHAT DID THE RESEARCHERS WANT TO UNDERSTAND?
Do Meta A/B tests deliver ads to different audiences, and can campaign settings reduce that divergence?
The researchers examine how balanced test groups are in their user characteristics. This matters because a comparison of two ads may involve changes in both content and the audience reached. The study also investigates settings that could make it easier to draw conclusions about advertising messages.
Explaining the context · DijitalPi commentary
DijitalPi commentary: The winner shown in a results dashboard does not necessarily answer the question your business wants to ask. Learning which option achieves a lower cost under ordinary campaign conditions is different from learning whether its message will work better with other audiences. Making that distinction before testing defines the limits of the lesson you can take from a winning ad.
02 / HOW WAS THE RESEARCH CONDUCTED?
How did they test the question?
Burtch, Moakler, Gordon, Zhang and Hill examined 3,204 Lift tests and 181,890 advertiser A/B tests on Meta. The Lift tests began on or after 15 March 2025 and finished by 10 June 2025, comprising approximately 15.5 billion user-test observations. The A/B tests began on or after 22 March 2025; data were collected on 15 June 2025 and comprised approximately 27 billion user-test observations. The analysis covered all participating users in Lift tests and a 10% random sample of A/B test impressions. Country coverage was not specified for the main sample. The main analysis checks audience balance in existing experiments rather than presenting a new experiment estimating advertising effectiveness.
Understanding the method · DijitalPi commentary
A Lift test compares users eligible to see advertising with a control group that cannot see it. An A/B test compares alternative campaign configurations without a no-ad control and counts only conversions attributed to advertising in its results. The researchers examined 14 characteristics, including age, gender, language and location, alongside a 72-dimensional numerical user representation used in Meta’s ad ranking, making 86 characteristics in total. A standardised mean difference (SMD) puts characteristics on a common scale to measure the gap between groups. The paper used a threshold of 0.2 for this measure.
The A/B sample was first narrowed to 46,912 tests with the same optimisation objective, target audience, budget and bid. The researchers then examined awareness tests using a single static image and approximately 1 impression per user. They extended the data window back two years as an additional check. One author also ran a 5-cell case study targeting US users under 65: each cell used a different static image, a reach objective, a frequency cap of 1 impression per user-week and Facebook news feed placement only. It ran for 4 days and accumulated more than 1.8 million impressions.
03 / RESEARCH FINDINGS
A/B ads often reached different audiences. Fewer tests supported the proposed fix.
The researchers detected no audience imbalance in Lift tests; only 0.16% of characteristic comparisons had standardised differences exceeding 0.2. In unfiltered A/B tests, the corresponding share was 22%. Imbalance remained after matching objectives, audiences, budgets and bids: the threshold was exceeded in 5.07% of comparisons across 612 awareness tests and 14.4% in conversion-objective tests. Only 3 tests in the main sample met the strictest conditions. These covered approximately 360,000 user-test pairs, with no detected imbalance and no standardised difference above 0.2. The additional 17 tests found by extending the window back two years showed a broadly similar pattern, although approximately 3% of comparisons for dimensions of the numerical user representation exceeded the threshold. No imbalance was detected in the case study either. Of the additional 17 tests, 8 showed statistically significant differences in click-through rates; the authors reported that the broad balance pattern also held in those tests. These findings are not estimates of sales or revenue growth.
Differences above the limit across all A/B tests
22%The share of user-characteristic comparisons with a standardised difference above 0.2; it is not the percentage of imbalanced tests or customers.
Differences above the limit in Lift tests
0.16%The share exceeding 0.2 on the same characteristic-difference measure; it does not measure advertising’s sales impact or the proportion of successful tests.
DijitalPi commentary: If you want to compare ad content alone, consider the authors’ recommendation as a separate test design: keep the reach objective, budget, bid strategy, schedule, target audience and manual placement identical, varying only the content. Include placement settings in the report because automatic placement can affect which audiences are reached. Comparing age and gender distributions is a useful starting point. The t-statistic threshold of 1.5 identified in 612 awareness tests is a warning for a statistic that assesses age or gender differences relative to variation within the groups; staying below it does not guarantee balance.
This proposal differs from an everyday conversion campaign. Only 3 tests in the main sample met the strict conditions, limiting confidence in the remedy; the filters also could not guarantee matching schedules, manual placements, music, destination links or calls to action. Advertisers cannot see the other user characteristics in test output. The 0.2 threshold also comes from another research context, and its suitability for advertising is uncertain. A winning A/B cell’s lower acquisition cost does not identify additional conversions that would not have occurred without advertising; Lift offers a different comparison for that question.
05 / CONCLUSION AND OPEN QUESTIONS
What did we learn, and what do we still not know?
This preprint shows that Meta A/B results can reflect the combined contribution of ad content and the audience reached. The authors interpret divergence as an intentional platform feature and stress that no A/B configuration can guarantee absolute balance. The March–June 2025 main data and limited number of strictly configured tests should not be generalised directly to Türkiye, Turkish-language advertising or other platforms. Transferring Lift findings to other contexts also depends on reaching the same audience. Peer review is not documented for the examined v1. Four of the five authors have Meta ties: Moakler, Zhang and Hill are regular employees and shareholders, while Burtch works part-time for eight hours per week. Gordon holds appointments at Northwestern and as an Amazon Scholar; the paper states that this work was performed at Northwestern and is not associated with Amazon. No separate external funding declaration was found; the study should not be presented as an independent external audit.
Our open question at DijitalPi: should your next Meta test inform a decision about a particular message, performance under current campaign conditions, or the additional impact of advertising?
We created these scenarios to make the topic concrete. They are not cases from the paper or measured client results.
EXAMPLE 01
E-commerce: extending the winning image to other campaigns
An illustrative homeware retailer compares a product photograph with an image showing the product in use and considers adopting the winner across all campaigns.
Does a lower customer acquisition cost mean the image communicates more effectively?
Open the recommendation for example 01+
Check objectives, budgets and audience differences first.
Record each group’s objective, budget, bid strategy, schedule and placement side by side. Examine age and gender distributions separately from customer acquisition cost. For a follow-up comparison of images alone, consider the authors’ recommendation of identical settings, a reach objective, a single static image and low impression frequency. This setup differs from an everyday conversion campaign and does not guarantee audience balance.
EXAMPLE 02
Local services: two ads reach different age groups
An illustrative home-cleaning business compares messages about saving time and service coverage, then notices that the ads reach different age distributions.
Does the result reflect message preference alone?
Open the recommendation for example 02+
Report the audience differences. Do not explain the result through the message alone.
Compare age and gender distributions across test groups and record whether automatic placement was used. If statistical comparisons of age and gender are available, treat the authors’ t-statistic threshold of 1.5 as a warning; it is not a percentage difference. Staying below it does not establish balance in characteristics absent from the report. The same relationship should not be assumed for a local Turkish-language campaign.
EXAMPLE 03
B2B software: did the winning campaign bring additional customers?
An illustrative business software provider interprets a lower customer acquisition cost in an A/B test as evidence that its advertising budget generated additional customers.
Can this result distinguish customers who would have arrived without advertising?
Open the recommendation for example 03+
Design a measurement with a no-ad control group to investigate additional impact.
First define the outcome, such as a completed demo request. Consider a Lift test plan that reports additional impact against a no-ad control separately from the conversions attributed to advertising in the A/B report. Do not present demo requests as signed contracts. This paper does not calculate your business’s additional customer acquisition, and transferring a Lift result to another setting also depends on the audience reached.
TRY IT WITH YOUR TEAM
Clarify the test question.
Write down whether your next test should compare ad content, campaign configurations or additional impact against a no-ad scenario. Plan consistent settings and audience balance checks for content comparisons, and a no-ad control for additional impact. This is a proposal that DijitalPi has not yet implemented.
Gordon Burtch, Robert Moakler, Brett R. Gordon, Poppy Zhang, Shawndra Hill Boston University Questrom School of Business (Burtch); Meta (Moakler, Zhang, Hill, Burtch); Northwestern University Kellogg School of Management (Gordon).
The academic work belongs to the researchers named above. This page contains DijitalPi's explanatory review and original business examples; it is not a full translation of the paper.
Cookie preferences