One question, three rankings
Same evidence, different ranks, different scores.
A collection holds 3 passages that are relevant to one question. Three fictional systems each return 5 passages. A 1 marks a relevant passage at that rank, a 0 an irrelevant one. All scores use k = 5.
| Ranking (ranks 1 to 5) | Recall@5 | Precision@5 | Reciprocal rank | DCG@5 | nDCG@5 |
|---|---|---|---|---|---|
| A: 1 0 1 0 0 | 0.667 | 0.400 | 1.000 | 1.500 | 0.704 |
| B: 1 1 0 0 0 | 0.667 | 0.400 | 1.000 | 1.631 | 0.765 |
| C: 0 0 1 0 0 | 0.333 | 0.200 | 0.333 | 0.500 | 0.235 |
A and B find the same two passages, so recall and precision cannot tell them apart. nDCG ranks B higher because its second hit arrives at rank 2 instead of rank 3. The ideal DCG here is 2.131: three relevant passages at ranks 1, 2 and 3. If A, B and C were three different questions, their mean reciprocal rank (MRR) would be (1 + 1 + 0.333) / 3 = 0.778.
The passages and labels are fictional. Real scores depend on the question set, the relevance labels, and the cutoff.
Interactive worked example
Calculate retrieval metrics for one query.
Change the relevance labels and cutoff to see why finding evidence and ranking it well are different. These five distinct passages are fictional. This example does not recalculate the published benchmark.
Include relevant passages missing from these five results. Use complete relevance judgments for this illustration.
2 relevant passages in the top 5; 3 known in the collection.
- Recall@5
- 66.7%
- Precision@5
- 40.0%
- nDCG@5 · binary
- 70.4%
- Reciprocal rank@5
- 1.000
DCG = 1.500; ideal DCG = 2.131. Recall and precision count hits; nDCG also rewards earlier hits. A single-query reciprocal rank is not MRR: MRR averages reciprocal ranks over an explicitly defined query set.
Formulas and limits of this example
- Recall@k = relevant passages in the top k / all known relevant passages.
- Precision@k = relevant passages in the top k / k. Every example has at least k distinct results.
- Binary DCG@k = the sum of relevance / log₂(rank + 1), with relevance 1 or 0.
- nDCG@k = DCG / ideal DCG. The ideal ranking places min(k, all known relevant passages) relevant results first.
- Reciprocal rank@k = 1 / the first relevant rank, or 0 when no relevant passage appears before the cutoff.
These are binary passage-relevance metrics. The published benchmark uses its sealed labels and source-anchor checks; its citation precision is a different measure. This example does not score generated answers, factual support, access boundaries, or freshness.
See the Stanford information retrieval reference (opens in a new tab) for ranked retrieval metrics. Calculations run in your browser without calling the retrieval API.
Formulas
What each metric measures, and what it misses.
| Metric | Formula | Answers | Blind spot |
|---|---|---|---|
| Recall@k | relevant passages in the top k / all known relevant passages | Did we find the evidence? | Ignores order inside the top k; needs a complete set of known relevant passages. |
| Precision@k | relevant passages in the top k / k | How much of the context is useful? | Capped when fewer than k relevant passages exist. |
| Reciprocal rank and MRR | 1 / rank of the first relevant passage (0 if none in the top k); MRR is the mean over all queries | How soon does the first useful passage appear? | Ignores every relevant passage after the first. |
| DCG@k | sum over ranks i of gain_i / log2(i + 1) | How much relevance appears, discounted by position? | Not comparable across queries without normalisation. |
| nDCG@k | DCG@k / ideal DCG@k, where the ideal ranking puts the most relevant passages first | How close is the ranking to the best possible one? | Depends on labels and gain choice; a perfect score can still miss evidence beyond k. |
With graded labels (for example 0, 1, 2, 3), DCG uses either the label itself or 2^label - 1 as the gain. Libraries differ, so record which variant you use before comparing numbers.
Interpreting results
What is a good nDCG or MRR score?
There is no universal threshold. The same system can score 0.9 on easy, well-labelled questions and 0.5 on ambiguous ones. A score is only meaningful against a fixed question set, fixed labels, and a stated cutoff.
- Compare like with like. Run every configuration on the same sealed questions and labels, and report k with every number (nDCG@10, not nDCG). nDCG@10 is nDCG computed over the top 10 results only.
- Agree the gate before the test. Write the minimum recall and ranking scores a configuration must reach before you look at the results.
- Keep failure cases visible. Averages hide the questions that matter most. Keep a list of queries where required evidence was missing.
- Score abstention separately. When no supported answer exists, the right retrieval result can be no evidence. Rank metrics do not reward that behaviour.
The EU AI Act retrieval benchmark shows one such gate on 80 fixed questions, with its limits stated next to each figure.
A high retrieval score does not show that the final answer is correct.
A common confusion
MRR is a metric. MMR is a re-ranking method.
Mean reciprocal rank (MRR) scores a ranking after the fact. Maximal marginal relevance (MMR), introduced by Carbonell and Goldstein in 1998, selects results that are relevant but not redundant with results already chosen. A RAG pipeline can use MMR to build its context and then be evaluated with MRR or nDCG.
What rank metrics do not cover
Retrieval quality is more than ranking.
A ranked list can score well and still be unusable in production. Test these properties separately.
- Freshness: is the passage from the version of the source that applies today?
- Boundaries: was the passage allowed for this caller and these filters?
- Citation support: does the cited passage actually support the statement built on it?
- Answer quality: evaluate the generated answer with its own method; retrieval metrics stop at the context.
To locate which stage fails, see the acceptance test in our RAG architecture guide. To see how a returned passage keeps its source, version and citation, open the reviewed source trace, or browse the evidence overview.
Further reading
Check the original definitions.
- Manning, Raghavan and Schütze (2008): Introduction to Information Retrieval, chapter 8 (opens in a new tab), Cambridge University Press, Evaluation in information retrieval, for ranked retrieval evaluation.
- Järvelin and Kekäläinen (2002): Cumulated gain-based evaluation of IR techniques (opens in a new tab), ACM Transactions on Information Systems, the origin of DCG and nDCG.
- Carbonell and Goldstein (1998): The use of MMR, diversity-based reranking (opens in a new tab), SIGIR 1998, for maximal marginal relevance.
- Discounted cumulative gain (opens in a new tab), for the gain variants used with graded relevance.
- Mean reciprocal rank (opens in a new tab), for the definition and a short example.