Can AI predict which content will get engagement?

01 / WHAT DID THE RESEARCHERS WANT TO UNDERSTAND?

Do the changes AI makes for higher engagement match the features of answers people actually vote for?

The researchers examine whether language models’ expectations about engagement align with vote rankings for real platform answers. They also test whether OMRA, the authors’ proposed method for masking and reconstructing passages the model has overexplained, reduces the measured mismatch. The underlying question is how closely a model’s preferred writing reflects actual user behaviour.

02 / HOW WAS THE RESEARCH CONDUCTED?

How did they test the question?

Researchers affiliated with Huazhong University of Science and Technology in China examined 1.17 million answers to 25,978 questions on Zhihu, Quora and Reddit from 2020-2026, covering 10 categories and 35 subcategories. The researchers examined existing answers. They also generated answers without publishing them and asked people to evaluate them. The controlled comparison used 3,600 real answers: one answer from each of four engagement levels for each of 900 questions. For human evaluation, 30 undergraduate and postgraduate students from a university assessed 400 paired items. The source does not specify the platform users’ country distribution or provide a language breakdown.

03 / RESEARCH FINDINGS

The model’s engagement expectations did not match real answers’ voting patterns

The researchers found that models added more claims, evidence and step-by-step justification as target engagement increased. In real answers, higher relative voting levels were more strongly positively associated with emotional appeal and expression, and weakly negatively associated with explicit reasoning. This observational relationship does not show that deleting explanations would increase votes. The researchers used four AI models to predict engagement rankings. Three performed worse than a basic method that converted text into numbers and compared them. The authors’ method reduced the average relative gap across four measures comparing how answer features are distributed by 54.4% compared with direct generation. In the human evaluation, directly generated answers appeared less human-written in 97.3% of comparisons, but their preference rate was 55.7%, with a range of 47.5-64.1% reflecting uncertainty in that rate. Because this range includes 50%, the preference difference cannot be distinguished from chance. The preference rate for the authors’ method was 62.4%, with a reported range of 53.6-69.9%. Generation guided by a real example achieved 58.8%, with a reported range of 50.5-66.9%. The authors describe the result favouring their method as supportive rather than statistically decisive because the ranges reported for these two methods overlap.

Average reduction in the measured gap

54.4%The authors’ method’s mean relative reduction across four measures comparing how answer features are distributed, compared with direct generation. It is not an increase in live platform engagement.

Preference for the authors’ method in human evaluation

62.4%The proportion of paired comparisons in which the answer revised using the authors’ method was preferred to a real answer. The range reflecting uncertainty in this rate is 53.6-69.9%. This is not a live voting or sales rate.

Source: §3 Methodology; §4.2 Evaluation Results, Tables 1-2; Appendix B Dataset ↗

04 / DIJITALPI'S EXPLANATION

How should we interpret these findings?

DijitalPi commentary: Check whether each explanation added by the model helps answer the reader’s question before treating it as an improvement. Recording a model’s engagement ranking before publication and comparing it with subsequent community votes could help assess whether its predictions are useful. This is our proposed assessment approach, not a marketing application tested in the paper.

Visibility, timing, author identity, ranking algorithms and community norms also influence vote rankings. Comparing answers within a question does not eliminate these effects. Length explains part of the measured gap but does not by itself explain the improvement achieved by the authors’ method. Scoring depends on a language model from a family also used for generation, and may misread sarcasm or culturally specific expression. A further model-based assessment of 200 outputs from the authors’ method reported unsupported additions in 6.5% and major changes in meaning in 4.0%. Check the facts in rewritten content. More natural wording is not enough.

05 / CONCLUSION AND OPEN QUESTIONS

What did we learn, and what do we still not know?

This study is a preprint, a research manuscript that has not yet undergone peer review. Generated answers were not published on the platforms. Engagement meant vote rankings among answers to the same question; clicks, sales and conversion gains were not measured. This preprint shows that the text features models favour for engagement differ from voting patterns in real question-and-answer communities. The authors’ method reduces the gap measured offline. It does not change the model’s underlying preferences or demonstrate higher engagement after publication. The findings are bounded by three platforms’ 2020-2026 data, model-based scoring and preference assessments from 30 university students. Without a language breakdown, applicability to Turkish content and Türkiye remains unestablished; the findings cannot be directly extended to branded posts or every sector. The reviewed v1 has not undergone peer review. All authors have university affiliations, and no company affiliation is stated. No funding or competing-interest declaration was found in the source.

Our open question at DijitalPi: will the answer your model expects to attract most engagement also receive most support in your Turkish customer community, and how will you assess the influence of visibility and publication timing?