Evaluation Metrics and Libraries for Agentic RAG Workflow
Learn to quantitatively measure your agent’s performance by creating evaluation datasets and applying modern, research-backed evaluation frameworks.
We'll cover the following...
In our last lesson, we learned the crucial, hands-on skills of debugging and refining our agent. We employed a qualitative approach, reading the agent’s trace to assess whether its reasoning seemed right. This is essential for fixing individual bugs.
But how do we prove that our changes are actually making the agent better across hundreds of different queries? How do we measure its performance objectively? To answer these questions, we need to move from the art of debugging to the science of evaluation. In this lesson, we will explore the critical field of agentic evaluation, learning how to use metrics and automated testing to measure our agent’s performance at scale.
The measurement problem: Why evaluating agents is hard
For years, the quality of language models has been measured using metrics such as BLEU and ROUGE. These metrics work by measuring the word-for-word overlap between a model’s generated answer and a single, pre-written “correct” answer.
For dynamic, agentic systems, these metrics are insufficient. Research has shown that traditional information retrieval (IR) metrics may not accurately measure the usefulness of retrieved documents in a RAG context, because an LLM consumes information differently than a human user. The quality of retrieval should be judged by its impact on the final task, not just by topical relevance.
This leads to several key challenges.
Multiple correct answers: An agent can generate a factually correct answer in many different ways. A word-overlap score would unfairly penalize a perfectly good answer just because it was phrased differently.
Dynamic data: Our agent uses tools to access live data from APIs. The “correct” answer to “Find me a recent paper on arXiv” will be different today than it was yesterday, making a static reference answer useless.
Process matters: A good agent not only gets the right answer, but also gets it efficiently and logically. These metrics tell us nothing about the quality of the agent’s reasoning process and its trace.
A framework for agent evaluation
To properly evaluate an agent, we need to look at it from multiple angles. We can group our evaluation into two main categories.
Component-level evaluation: How good are the individual parts?
End-to-end evaluation: How good is the final answer?
Within these categories, we can measure several dimensions:
Task success: Did the agent ultimately accomplish the user’s goal? This can be a simple pass/fail or a more nuanced score based on metrics like F1 and exact match (EM).
Efficiency and cost: How efficient was the agent? This includes measuring latency (total time taken) and the number of LLM calls or tokens consumed.
Reasoning quality (trace evaluation): Was the agent’s reasoning process logical and efficient? This involves analyzing the intermediate steps, such as the number of reasoning-retrieval iterations.
To add more granularity, we can categorize failures based on where they occur in the RAG process. Errors can arise in the retrieval stage (e.g., retrieving incomplete or irrelevant information) or in the generation stage (e.g., generating an inaccurate or off-topic response). A comprehensive evaluation system should be able to pinpoint where a failure occurred in the chain.
Building our “golden dataset”: The exam paper
To evaluate our system, we need a standardized test where every question can be answered from our knowledge base. We will now create a golden_dataset.jsonl file where each question and its ground_truth answer are derived directly from the “RAG and Beyond” survey by Zhao et al. and the “Evaluating Retrieval Quality” paper by Salemi and Zamani.