What happened to news publishers’ visit figures after they blocked AI crawlers? SimilarWeb and Semrush data put weekly visits about 7% lower during the 6 weeks after blocking. This estimate suggests that blocking AI bots may be associated with fewer human visits to the site.
ACADEMIC WORK REVIEWED
Strategic Response of News Publishers to Generative AI
ResearchersHangcheng Zhao, Ron Berman
arXiv · 15 April 2026 · v4, not peer reviewed · Preprint · Source language: English
English review: 28 October 2026 · The source date appears in the citation above.
Editorial review draft
The source has been checked; the review still carries its original editorial status. DijitalPi did not conduct a customer experiment for this paper.
LET'S READ THE RESEARCH TOGETHER
We first explain the researchers' question, method and findings. We then discuss how to interpret the results, clearly separating DijitalPi's commentary from the source.
01 / WHAT DID THE RESEARCHERS WANT TO UNDERSTAND?
How does blocking AI crawlers affect visits to news publishers’ websites?
The researchers examine how news publishers respond to generative AI through crawler restrictions, changes to content presentation and recruitment decisions. The central decision concerns how restricting access to content relates to traffic reaching the publisher’s own website.
Explaining the context · DijitalPi commentary
DijitalPi commentary: Managing the conditions under which content is used and attracting website visits are separate objectives. This study can inform the traffic dimension of an access decision. Looking only at visits without first specifying the objective could obscure why a restriction was chosen for content rights or licensing reasons.
02 / HOW WAS THE RESEARCH CONDUCTED?
How did they test the question?
Hangcheng Zhao and Ron Berman studied 30 large newspaper publishers selected from 6,315 websites appearing in both SimilarWeb and Revelio. Semrush and Comscore analyses extended coverage to the 500 news publishers with the highest Comscore traffic among matched sites. SimilarWeb supplied worldwide daily desktop and mobile visit estimates from 1 January 2019 to 28 February 2026. Comscore supplied US household desktop browsing data for 2022-2024. Strategic responses were examined in the context of November 2022-May 2024, while the main blocking analysis was restricted to before May 2024. The design was observational, without random assignment.
Understanding the method · DijitalPi commentary
robots.txt is a file that tells automated crawlers which access rules apply. The researchers compared traffic changes at publishers introducing blocks at different times with changes at publishers that had not yet blocked or never blocked. Blocking meant an explicit restriction for a particular AI crawler in robots.txt. Observations after a publisher subsequently removed its block were excluded. The main comparison covered 12 weeks before blocking and 6 weeks afterwards. The outcome was weekly visits, analysed on a logarithmic scale, a transformation that helps compare proportional changes across websites of different sizes.
SimilarWeb and Semrush results relied on third-party traffic measurements rather than publishers’ own analytics records. Comscore provided a separate panel of human browsing. The researchers used alternative analyses, checks that treated dates without a block as if blocking had started then and tests of whether earlier traffic changes predicted blocking decisions. They compared changes in content presentation, measured through the HTML elements making up pages, with the top 100 retail websites. Revelio job advertisements measured demand for workers rather than completed recruitment.
03 / RESEARCH FINDINGS
Short-term traffic estimates were negative, with precision varying by source and publisher size
The researchers reported approximately 7% fewer weekly visits in SimilarWeb and Semrush measurements during the 6 weeks after blocking. A confidence interval is a range showing uncertainty around an estimate. In Table 2, the SimilarWeb interval excludes zero. The Semrush interval includes zero, so it does not rule out no change in visits. The approximately 7% decline estimated using Comscore’s human browsing panel is also statistically insignificant. The three sources therefore do not provide equally conclusive results. Estimates are negative and significant for the top 50 publishers, negative but insignificant for ranks 51-100, and no longer negative for ranks 101-500. Longer-term estimates remain negative but become statistically insignificant after approximately 20 weeks. Not all alternative analyses produce significant results either.
Estimated weekly visits after blocking
Approximately 7% declineThe reported SimilarWeb and Semrush estimate for the following 6 weeks. The Semrush confidence interval includes zero. This is neither a sales loss nor a forecast for your business.
Limit of the longer-term estimate
Approximately 20 weeksBeyond this point, negative estimates are statistically insignificant. This does not establish that traffic returned to its previous level.
DijitalPi commentary: If you plan an access change, monitor individual traffic channels alongside total visits and assess the decision against more than short-term traffic objectives. The authors report that before May 2024, the decline was concentrated in direct visits while organic search referrals remained broadly stable. Their suggestion that fewer source mentions in AI responses could reduce brand exposure is a proposed explanation, not a directly measured mechanism. Visit data alone cannot establish it.
Publishers responded through more than blocking: after November 2022, advertising and targeting technologies increased by approximately 50%, interactive elements by 68.1%, and site framework and layout elements by approximately 70.2%, relative to retail websites. There was no increase in new text or article URLs and no clear evidence of an abrupt decline in editorial job advertisements. These findings do not show that adding interactive elements generates traffic or that employee numbers were maintained. Job advertisements do not measure employment, and changes in page elements alone do not establish content quality.
05 / CONCLUSION AND OPEN QUESTIONS
What did we learn, and what do we still not know?
This study is a preprint, a research manuscript that has not yet undergone peer review. It measured visits, not sales or revenue. This preprint suggests that crawler blocking and short-term declines in visits can occur together among large news publishers; its observational design does not establish definitive causation. The finding that earlier traffic changes did not predict blocking, alongside other checks, supports the analysis, but measurement uncertainty remains for smaller sites. The main analysis concerns the period before May 2024, and traffic data do not capture consumption occurring entirely within AI interfaces. It provides no validated effect for Turkish content or businesses in Türkiye. The reviewed text is v4, dated 15 April 2026, and has not undergone peer review. Partial financial support was received, but the funder is unknown because its name was removed for blind review. The authors are university academics and declare no affiliations with organisations having financial or non-financial interests in the subject.
Our open question at DijitalPi: which objective will determine your crawler access decision: visits, content rights or licensing? How will you measure that objective separately?
We created these scenarios to make the topic concrete. They are not cases from the paper or measured client results.
EXAMPLE 01
News publishing: access and licensing decisions
An illustrative news publisher is considering restricting AI crawlers before content licensing negotiations.
If visits fall, does that mean blocking has failed?
Open the recommendation for example 01+
Assess traffic objectives separately from content rights objectives
Record the crawler rule to be changed, the implementation date and the licensing objective in advance. Monitor weekly visits by channel and note other website changes during the same period. The paper discusses publishers potentially accepting short-term traffic losses to strengthen their negotiating position. It does not measure whether that strategy succeeds.
EXAMPLE 02
E-commerce: deciding from news publisher evidence
An illustrative homeware retailer is considering removing an existing crawler block after reading the news publisher findings.
Will opening access increase our visits too?
Open the recommendation for example 02+
Build your own measurement plan without treating the paper as a forecast
Document existing access rules and weekly visits before any change. If you make a change, also record the campaign schedule and track visits and orders separately. The study does not test the effect of removing blocks on retail websites or support transferring the news publisher estimate to your shop.
EXAMPLE 03
B2B software: a decline in direct visits
An illustrative B2B software company observes fewer direct visits after changing crawler access.
Does the decline show that AI mentions our brand less often?
Open the recommendation for example 03+
Measure traffic changes and brand mentions separately
Examine direct visits and organic search trends separately. Also record brand mentions in AI responses to relevant product questions, while tracking sales enquiries as a distinct outcome. The paper proposes reduced brand exposure as a possible explanation without measuring that mechanism directly, so a decline in visits alone cannot confirm it.
TRY IT WITH YOUR TEAM
State why you want to change crawler access
Before changing access, specify your objective, the traffic channels to monitor and acceptable conditions for content rights. Record concurrent campaign and website changes when evaluating results. This is a measurement proposal that DijitalPi has not yet implemented.
PiScan checks, across five layers, whether AI crawlers can read and cite a page on your site. It does not measure how an access decision affects visits or sales. Explore PiScan ↗
The academic work belongs to the researchers named above. This page contains DijitalPi's explanatory review and original business examples; it is not a full translation of the paper.
Cookie preferences