This article was written by Research Engineer Myriam Robbana of the Artificial Intelligence Lab within Berger-Levrault’s BL Research team.
AI agents are increasingly used to retrieve information and reason over knowledge bases. As these systems become more complex, evaluating only their final answers is no longer enough: a correct-looking answer can hide failures in the steps that produced it.
1. Fast, confident, but possibly wrong
A maintenance technician asks an AI assistant where to find the annual servicing procedure for a specific robot. A few seconds later, a clear, well-structured, confident answer appears, with a couple of document references attached. The technician acts on it.
Nothing in that answer tells them whether the process behind it worked as intended, or whether the response is actually relevant and well-grounded. More specifically, they cannot tell whether the assistant queried the right document collection or whether the passages it retrieved were actually about the right machine. The answer looks the same whether the pipeline worked perfectly or failed at the first step.
For the team developing and maintaining such an agent, this creates a different problem: when an answer is wrong, where did the failure occur? Was the request routed to the wrong agent or collection? Did retrieval return irrelevant passages? Did the model misinterpret correct evidence? Without visibility into these intermediate steps, diagnosing the system from its final answers alone is difficult.
This is the problem we faced with Athena, a conversational assistant designed to help users leverage their knowledge base and support them in their daily work. Athena relies on a multi-agent architecture: an orchestrator interprets the request, selects a specialised agent, and that agent queries technical documentation through a Retrieval-Augmented Generation (RAG) pipeline or calls tools exposed through MCP.
The path depends on the type of request. For RAG-based questions, the answer goes through three stages.
- Routing: the orchestrator decides who handles the request, which agent, which collection, and which MCP tool.
- Retrieval: the selected agent searches the relevant collection and retrieves the passages most likely to contain the answer.
- Generation: the model produces the final answer based on the retrieved passages and attaches its citations.
For non-RAG questions, retrieval is skipped: the request goes through routing and generation only.
The question we set out to answer was:
How do you automatically evaluate each step of an AI agent’s reasoning, when no ground truth exists?
2. A wrong answer, but wrong where?
In an agentic system, failures propagate from one stage to the next, each degrading the quality and truthfulness of the final answer. Routing comes first: if the request is sent to an irrelevant agent, everything downstream is compromised, whether the answer is meant to come from documents or from a tool.
Two paths can then fail independently. On the documentary path, the request reaches the wrong collection or the retrieved passages are irrelevant, which leads to wrong/incomplete answers, or to refusals when the information is in fact present in the corpus. On the tool path, the orchestrator may select the wrong MCP tool, or call the right one with incorrect or incomplete arguments. Unlike an empty retrieval, such a call succeeds and returns a plausible result, which makes the error much harder to detect. The tool may also return nothing, and the system must then distinguish an unavailable resource from genuinely absent information.
Generation is common to both paths. Even when everything upstream is correct, the model may misinterpret the evidence it was given, introduce unsupported information, or cite documents that do not support its claims.
The figure below traces the three stages an answer goes through, with a possible failure at each one.
Each of these failures produces the same thing: a fluent, confident answer. It is therefore tempting to evaluate an AI assistant as a single generative model: ask a question, inspect the final answer, and assign a score. However, this only tells the team that something went wrong, not where. A final answer can hide failures that occurred earlier in the pipeline.
For this reason, we decompose the evaluation into three components, each targeting a distinct stage of the pipeline, with its own metrics and validation protocol:
- Routing: Did the system select the appropriate agent, tools, collections, and arguments?
- Retrieval: Are the retrieved passages relevant and correctly ranked?
- Generation: Is the answer coherent and grounded in the retrieved passages? Are citations correct? Does the model abstain when the available context does not support a conclusion?
3. The constraint that changes everything: No ground truth
Most of the strongest evaluation approaches in the literature share one prerequisite: a reference set built in advance, stating which documents are relevant to which questions and which answers are correct. This is not incidental. Retrieval metrics such as recall@k or nDCG, and correctness metrics such as exact match or F1, are defined relative to a label. Fine-tuned evaluators need thousands of annotated examples to learn a scoring behaviour. And even reference-free approaches must prove their judge is reliable, which is done by correlating its scores with human judgements on an annotated subset.
However, that prerequisite does not survive contact with a multi-client product. Each client brings its own documents. The corpus therefore differs from one client to another, both in content and structure, and continuously evolves as documents are updated and new clients are onboarded. Building and maintaining a reference set for each client is not realistic.
This constraint also rules out some of the most efficient approaches at inference time, such as fine-tuning a dedicated evaluator or training a lightweight textual entailment model. Both require substantial annotated data and would have to be retrained as each client’s documents evolve. Without ground truth, we cannot rely on approaches that need a predefined reference dataset.
We also explored the obvious workaround: generating synthetic question-answer pairs from client documents to serve as a pseudo-reference. This was tested internally and set aside because the generated questions drifted too far from what users actually ask, both in phrasing and in underlying need. More importantly, synthetic generation introduces its own hallucination and bias risks, inherited from the model producing the pseudo-reference, which would then themselves need to be evaluated.
This leaves LLM-as-a-Judge as the most practical approach: it requires no prior annotation and allows a new client corpus to be evaluated without training a dedicated evaluator for each one. The challenge then shifts from finding a reference-based evaluator to determining how reliably the judge itself can evaluate the system.
An LLM judge is, however, an imperfect instrument, and its outputs depend heavily on how it is prompted. Using one therefore requires validating the judge prompt first, to ensure it produces consistent and meaningful assessments before its scores can be relied upon.
4. Judge once, compute everything else
4.1. Evaluating the retrieval
4.1.1. Defining “Relevance”
Everything hinges on what “relevant” means, so it had to be defined before anything could be judged. This is why we used this definition of “relevance”:
“A passage is relevant if it can serve as evidence, source or material for a correct answer”, which we broke down into three criteria: the right target (the equipment or object the query is about), the right intent (find a document, follow a procedure, locate a part, diagnose a fault), and enough usable content to actually contribute.
4.1.2. From relevance scores to retrieval metrics
Existing frameworks issue a separate LLM call for every metric they compute. Evaluation cost grows with the number of metrics, and every metric inherits the judge’s variability.
We therefore rethought the design. The retrieval component illustrates this principle particularly clearly:
- The judge is called once per (query, chunk retrieved) pair and answers a single question: how relevant is this passage to this query?
- That score is stored.
- Every retrieval metric is then computed deterministically from the stored scores: mean context relevance, Precision@K, NDCG@K, …
The judge is never asked a second question. Precision@K is not an LLM verdict, it is a count. NDCG is a formula. Adding a new metric requires no additional LLM call or judge validation.
Thus, all seven metrics are computed deterministically from the stored scores, without any additional judge call:
| Metric | What it measures |
|---|---|
| Context Relevance | Average relevance of all retrieved chunks |
| Maximum relevance | Score of the best retrieved chunk |
| Mean relevance of cited chunks | Average score of the chunks cited in the answer |
| Precision@K | Proportion of relevant chunks placed first |
| NDCG | Ranking quality: are the relevant chunks placed first? |
| Cited chunk precision | Chunks both relevant and cited, over all retrieved chunks |
| Cited chunk Recall | Chunks both relevant and cited, over relevant retrieved chunks |
The consequence matters more than the saving. The entire evaluation rests on one subjective judgment: the relevance score. Validate that, and everything downstream is arithmetic. Instead of validating a dozen metrics, we validate one.
4.2. Evaluating the generation
The same architecture applies to generation. The judge is called once per (paragraph, cited chunk) pair in the response, to produce a support verdict. Those verdicts fill a matrix, from which faithfulness and citation metrics are derived without any further call.
Therefore, each metric below comes out of that matrix:
| Metric | What it measures |
|---|---|
| Faithfulness to cited references | Whether the claims in the answer are faithful to the sources they cite |
| Citation correctness | Does the model correctly interpret the information contained in each cited chunk? |
| Citation Recall | Does the model cite its sources often enough to justify the information it puts forward? |
| Citation precision | Are the citations actually necessary and relevant? |
| Question understanding | Does the answer correctly address the user’s intent, without off-topic content? |
| Coherence | Is the answer structured, logical and understandable? |
| Completeness | Does the answer cover every element that was asked for? |
| Language compliance | Is the generated answer in the same language as the user’s question? |
| Appropriate abstention | Does the model recognise when the available information is insufficient? |
4.3. Evaluating the routing
For routing, one call per conversation covers the relevance of the collections and tools mobilised and the correctness of the arguments passed.
The five routing metrics follow:
| Metric | What it measures |
|---|---|
| Collection OK rate | Relevance of the document collections that were queried |
| Tool OK rate | Relevance of the MCP tools that were invoked |
| Args OK rate | Correctness of the arguments passed to the calls |
| Missing resource rate | Relevant resources that were never called |
| Refined query used rate | Use of the reformulated query |
5. Who judges the judge?
The same principle applies to all three components: use human annotations to validate the semantic judgments made by the LLM. The goal is always the same: make sure that the judgments used to diagnose the system match human assessment.
5.1. A human reference
For retrieval, two annotators scored the same conversations independently, on a fine-grained scale and a coarser one. Agreement, measured with quadratic weighted Cohen’s kappa, was substantial on both scales. Scale granularity did not change the consensus between humans.
Generation and routing were annotated the same way, against a fixed grid covering answer quality, abstention, citation support, and the coherence of the judge’s own justification.
5.2. LLM-Judge vs Human
We then compared the judge’s outputs to that reference. For retrieval, where a single relevance score carries the whole component, we tested twelve versions of the prompt and kept the one that stayed closest to the human annotations. As for Generation and routing we compared not only the individual judgments of the LLM but also the final metrics. All prompts are zero-shot (the judge is given the criteria but no worked examples), so it stays general instead of specialising on a handful of cases.
6. Where this leaves us
The comparison showed consistent results across all three components. The annotators reached substantial agreement, and the judge’s verdicts and derived metrics were consistent with the human reference. The validated evaluation approach can then be reused across corpora and adapted to new client contexts, allowing teams to systematically diagnose their agents.
Evaluating an AI assistant means more than checking its final answer: a failure can come from routing, retrieval, or generation, and there is no ground truth to tell us where.
We addressed this by evaluating these three components with a validated LLM-as-a-Judge, following one key principle: judge once, compute everything else deterministically. This makes the evaluation scalable across client corpora, while limiting the need for repeated manual evaluation.
The framework does not make the assistant more reliable on its own, but it makes its failures locatable: routing, retrieval or generation, rather than a single black-box answer.
