Measured retrieval evidence
How we evaluated retrieval quality on 80 EU AI Act queries.
A citation-sensitive RAG system needs more than recall. We test ranking, source anchors, citation focus, abstention, and scope violations against one frozen corpus and fixture set.
Direct answer
Which RAG evaluation metrics matter?
Use recall and rank quality to measure retrieval, source-anchor recall and citation precision to measure source alignment, and explicit abstention tests to catch unsupported queries. Track boundary violations separately. A high recall score cannot prove that citations are focused, that unsupported queries abstain, or that the result stayed inside its authorized corpus.
Evaluation framework
One corpus. One sealed fixture set. Several failure modes.
The benchmark uses 1,160 indexed document rows and 80 fixed queries. Fifty fixtures calibrate the abstention threshold; 30 remain in validation. The set contains 40 semantic queries, 20 exact references, 10 multi-evidence requests, and 10 unsupported queries. Ranking and citation metrics average the 70 answerable queries; abstention accuracy covers the 10 unsupported queries. The calibration and validation split serves threshold selection, not the denominator of every metric.2
- 40 Semantic queries
- 20 Exact references
- 10 Multi-evidence requests
- 10 Unsupported queries
- Step 1
Freeze the boundary
The corpus rows, fixtures, retrieval configuration, and checksums are fixed before comparison.
- Step 2
Measure ranked evidence
Each fixture identifies relevant chunks and required official source anchors at a cutoff of ten.
- Step 3
Test refusal behavior
Unsupported queries must abstain. Every returned passage must remain inside the enforced source scope.
Production endpoint result
The selected configuration passed the sealed gate.
These values come from 80 responses recorded against the deployed private-beta endpoint on 2026-08-21. This is a frozen v1 evaluation, not a measurement of the current corpus or live source freshness.1
- Limit
Finding evidence is not the same as ranking it well.
nDCG@10
82.0%1
Whether relevant evidence appeared near the top, with stronger credit for better ordering.
LimitRewards order inside the top ten only.
- Limit
Looks only at the first relevant passage.
Source-anchor recall@10
98.3%1
Whether the expected official article or annex anchor appeared in the result set.
LimitChecks that the anchor is present, not its rank.
Citation precision@10
53.4%1
What share of retrieved passages matched a required citation anchor. It measures focus, not factual correctness.
LimitLower than recall: room to improve focus. It does not prove legal correctness.
Abstention accuracy
100%1
Whether unsupported queries returned an explicit abstention instead of unrelated context.
LimitCovers only the 10 unsupported fixtures in this set.
Boundary violations
01
Returned passages outside the enforced source scope. Tracked separately from quality scores.
LimitApplies to this frozen corpus and its enforced scope.
Recall alone is not enough.
These metrics answer different questions. CorpusMesh keeps them separate instead of collapsing retrieval quality into one score.
Ranking and citation metrics average the 70 answerable queries. Abstention accuracy covers the 10 unsupported queries. Boundary violations count all 80 recorded responses.
The score profile is intentionally visible: citation precision is lower than recall. That is useful evidence for improving focus, not a number to hide.
Request beta accessConfiguration selection
The alternative did not pass.
Both embedding candidates used the same 1,160 rows and 80-query fixture set. Gemini embedding 2 at 768 dimensions cleared the gate. Voyage 4 at 1024 dimensions did not.
| Configuration | Gate | Recall@10 | nDCG@10 | MRR@10 | Citation precision |
|---|---|---|---|---|---|
| gemini-embedding-2 · 768D | Passed | 96.2% | 82.0% | 78.9% | 53.4% |
| voyage-4 · 1024D | Not selected | 92.6% | 69.9% | 64.7% | 49.1% |
Worked query
Article 6 and Annex III must appear together.
The fixed source-trace gate asks how Article 6 classification depends on Annex III. The production trace ranked Article 6 first and Annex III at position 3.3
- Article 6 rank
- 1
- Annex III rank
- 3
- Fixture split
- 50 calibration / 30 validation
- Runtime commit
57c28e321dbd
Reproducibility and limits
What the benchmark proves, and what it does not.
The fixture, chunk manifest, runtime report, and deployed commit are recorded so the result can be tied to one evaluated system state. These identifiers support provenance; this page does not provide the complete fixture and corpus artifacts needed for an independent replay.
- Fixture set SHA-256
c1780012a679b35b8d43e32bd8be4149af391be7cd971a66e54d2f348fa65b73- Chunk manifest SHA-256
2d73f00da7bd0cfbf942c26e96dbe27bbcf9161f428dd7985d6d669581e74924- Runtime report SHA-256
052cbcd7621f56e6d37b7addd7c931234af2d0f1422948bc3c29210eb2500c7f
Known limits
- This benchmark covers one English EU AI Act corpus and its fixed fixture set.
- Citation precision measures required-anchor focus. It does not prove legal correctness.
- The results do not establish performance for another domain, language, corpus version, or customer workload.
- The public evidence does not expose a public retrieval endpoint or legal advice service.
Your evaluation worksheet
Agree the acceptance criteria before the pilot.
Use this public benchmark to inspect the method. Define a separate gate for your own sources and application, with an owner and a pass condition for each row.
| Check | Record before evaluation | Evidence to review |
|---|---|---|
| Source scope | Authorities, documents, version, languages, jurisdictions, and exclusions | Every returned passage stays within the approved boundary. |
| Answerable queries | Representative questions and expected article or section anchors | Recall, ranking, and citation focus, with query counts and the chosen cutoff. |
| Unsupported queries | Questions outside the source scope or missing required evidence | Explicit abstention rather than unrelated context. |
| Source changes | Check cadence, candidate review, activation criteria, and recovery owner | A reviewed version, regression results, and a recoverable prior version. |
| Application outcome | Your model, prompts, user workflow, and acceptance criteria | Final-answer quality and usability measured separately from retrieval. |
Citation precision here is 53.4%: high recall still leaves room to improve the focus of retrieved passages. The worked query is one successful trace, not a complete account of every failure. The published benchmark uses ten results; the prepared beta plan caps a request at five, so evaluate your workload at its actual limit.
Review the freshness method and the published beta scope and price before defining your pilot.
Interactive worked example
Calculate retrieval metrics for one query.
Change the relevance labels and cutoff to see why finding evidence and ranking it well are different. These five distinct passages are fictional. This example does not recalculate the published benchmark.
Include relevant passages missing from these five results. Use complete relevance judgments for this illustration.
2 relevant passages in the top 5; 3 known in the collection.
- Recall@5
- 66.7%
- Precision@5
- 40.0%
- nDCG@5 · binary
- 70.4%
- Reciprocal rank@5
- 1.000
DCG = 1.500; ideal DCG = 2.131. Recall and precision count hits; nDCG also rewards earlier hits. A single-query reciprocal rank is not MRR: MRR averages reciprocal ranks over an explicitly defined query set.
Formulas and limits of this example
- Recall@k = relevant passages in the top k / all known relevant passages.
- Precision@k = relevant passages in the top k / k. Every example has at least k distinct results.
- Binary DCG@k = the sum of relevance / log₂(rank + 1), with relevance 1 or 0.
- nDCG@k = DCG / ideal DCG. The ideal ranking places min(k, all known relevant passages) relevant results first.
- Reciprocal rank@k = 1 / the first relevant rank, or 0 when no relevant passage appears before the cutoff.
These are binary passage-relevance metrics. The published benchmark uses its sealed labels and source-anchor checks; its citation precision is a different measure. This example does not score generated answers, factual support, access boundaries, or freshness.
See the Stanford information retrieval reference (opens in a new tab) for ranked retrieval metrics. Calculations run in your browser without calling the retrieval API.
Where each figure comes from.
- Runtime endpoint reportInspect the RAG evaluation
Recorded 2026-08-21 on the deployed private-beta endpoint, 80 responses.
Deployed commit 57c28e321dbddcad4eb57006d4015a5c1e0e204f
Report SHA-256 052cbcd7621f56e6d37b7addd7c931234af2d0f1422948bc3c29210eb2500c7f
- Sealed fixture set and corpus
80 fixed queries on 1,160 document rows, cutoff 10. Split 50 calibration / 30 validation.
Fixture set SHA-256 c1780012a679b35b8d43e32bd8be4149af391be7cd971a66e54d2f348fa65b73
Chunk manifest SHA-256 2d73f00da7bd0cfbf942c26e96dbe27bbcf9161f428dd7985d6d669581e74924
- Published source traceInspect the cited EU AI Act trace
Reviewed 2026-08-21 by CorpusMesh product and source review. Corpus version 2026-08-21.1, 5 of 10 results shown.
Public trace SHA-256 88821ad972ee9c05ed2d56e6f125baa0000c9f9abc4515f34dd82db3ef61fc17
- Official sourceOpen the regulation on EUR-Lex (opens in a new tab)
EUR-Lex, Regulation (EU) 2024/1689, verified 2026-08-21.
Evaluate your workload
Test the same retrieval contract against the evidence your product needs.
Tell us the sources, jurisdictions, query types, and failure modes that matter. We will define the fixture boundary before discussing a production knowledge base.