INTERNATIONAL RESEARCH / ACADEMIC PUBLICATION

Can we trust AI when it provides citations?

How did ALCE researchers evaluate the sourced answers and what did they find? We explain the meaning of the findings and compare the claim with its basis in four examples.

ACADEMIC WORK REVIEWED

Enabling Large Language Models to Generate Text with Citations

ResearchersTianyu Gao, Howard Yen, Jiatong Yu and Danqi Chen · Princeton University

EMNLP 2023 · conference paper · Source language: English

Open the original publication

ENGLISH REVIEW AND COMMENTARY: DIJITALPI

We explain the paper in plain English, discuss what it may mean for organisations and identify our own examples separately.

Read the explanation Go to 4 business examples

English review: 9 September 2026 · The source date appears in the citation above.

Editorial review draft

The source has been checked; the review still carries its original editorial status. DijitalPi did not conduct a customer experiment for this paper.

LET'S READ THE RESEARCH TOGETHER

We first explain the researchers' question, method and findings. We then discuss how to interpret the results, clearly separating DijitalPi's commentary from the source.

01 / WHAT DID THE RESEARCHERS WANT TO UNDERSTAND?

Does putting a source next to an answer make it auditable?

Tianyu Gao and colleagues developed the ALCE evaluation framework to reproducibly measure the quality of long AI answers citing sources.

Explaining the context · DijitalPi commentary

Citation means showing the source on which information is based. One of the things that makes an answer seem trustworthy are the links at the end. When the reader sees these links, he may think that the information has been checked. However, actually opening the link, accessing the relevant section, and examining whether it supports the claim in the sentence are separate tasks.

Let's think about it with an everyday example: Adding the names of ten books to the end of a report does not mean that these books confirm every sentence in the report. The reader must be able to establish the relationship between the sentence and the basis. This relationship is the starting point for understanding the problem the research is interested in.

02 / HOW WAS THE RESEARCH CONDUCTED?

How did they test the question?

It evaluates fluency, answer accuracy and citation quality separately in ALCE, ASQA, QAMPARI and ELI5 datasets. Automated source checking uses a language inference model that evaluates whether the given text supports the claim.

Understanding the method · DijitalPi commentary

Fluency describes how easily the answer can be read. Accuracy asks whether the information is correct, while citation quality asks whether the cited document actually supports it. An answer can perform well in one area and poorly in another.

It is useful to think in two ways in citation control. First, we see if there is any basis for the claims in the answer. Then we examine whether the added sources contribute to the relevant claim. Thus, we can notice both the unsupported sentence and the link added just to make it crowded.

03 / RESEARCH FINDINGS

Even with fluent answers, source support may be lacking.

Approximately half of the ELI5 answers in the study for the ChatGPT and GPT-4 base schemes are not fully supported by the text shown. Producing the answer first and adding the source later also gives poorer citation quality compared to answer accuracy.

Source: Purpose and main findings of the research, PDF pages 1–2 ↗

04 / DIJITALPI'S EXPLANATION

How should we interpret these findings?

Read the phrase "not fully supported" carefully. A long answer may contain four claims: three clearly supported by the source and one without a clear basis. In that illustrative case, the answer is not fully supported, but that does not mean all four claims are false.

Likewise, information not found in the source may be true elsewhere. The problem for the reader is that it cannot be verified by the link shown. For this reason, the notes "incorrect information", "source not found" and "only part of it is supported" should not be used interchangeably.

In a customer assistant, this distinction changes your decision. If the information is incorrect, you correct the answer. If the information is correct but the document is incorrect, you correct the attribution. If no basis can be found, you ask for confirmation from the relevant person. These are our interpretation of the application; a single overall trust score can leave these different jobs invisible.

05 / CONCLUSION AND OPEN QUESTIONS

What did we learn, and what do we still not know?

ALCE shows that generating answers with reliable citations remains an open problem. Its automated evaluator also has limits, especially when distinguishing partial support. These findings come from the study's 2023 experimental setups and do not represent the error rate of today's products.

How should the same checks be performed when old and new versions of Turkish documents coexist? We make this question concrete in the four exercises below. They are not the article's test data or measured model errors; they are examples prepared by DijitalPi.

Return to the researchers' original publication

Purpose and main findings of the research, PDF pages 1–2 Citation evaluation and limitations, original publication

DIJITALPI'S APPLICATION COMMENTARY

What could this look like in your organisation?

We created these scenarios to make the topic concrete. They are not cases from the paper or measured client results.

EXAMPLE 01

The claim appears in the cited source

An assistant says that the BAS-01 programme contains four sessions and cites the exact section where the session count appears.

Does the citation support this narrow claim?

Open the recommendation for example 01

Yes, for the session count alone.

The cited passage states the number directly, and the sentence adds no extra condition. Supporting this narrow claim does not verify every other fact about the programme.

EXAMPLE 02

Only part of the sentence is supported

The assistant says that BAS-01 has four sessions and awards a certificate, but the cited passage mentions only the session count.

Can the whole sentence be treated as supported?

Open the recommendation for example 02

Separate the sentence into two claims.

The source supports the session count but says nothing about a certificate. A current document or confirmation from the organisation is needed for the second claim.

EXAMPLE 03

The fact is right, but the citation is wrong

The assistant gives the correct current capacity of six participants but cites a section that only describes support hours.

Is a correct answer enough when its citation points elsewhere?

Open the recommendation for example 03

Correct the attribution as well as checking the fact.

Reaching the right answer does not make the cited passage relevant. Link the claim to the section that actually states the current capacity.

EXAMPLE 04

One source supports the claim, but current records conflict

One current document says that a session lasts 90 minutes, while another equally authoritative document from the same date says 120 minutes.

Does one supporting citation settle the decision?

Open the recommendation for example 04

Show the conflict and request confirmation.

A citation can support the sentence locally without resolving a conflict between valid sources. Present both durations and ask the programme owner to confirm which one applies.

Source and ownership

Tianyu Gao, Howard Yen, Jiatong Yu and Danqi Chen · Princeton University

EMNLP 2023 · conference paper. Open the original publication

The academic work belongs to the researchers named above. This page contains DijitalPi's explanatory review and original business examples; it is not a full translation of the paper.

← Return to all research articles