RAG evaluation helps you determine whether a document-based AI assistant retrieves the right evidence and uses it correctly. Retrieval-augmented generation can connect a model to your knowledge base, but that connection does not automatically make every answer accurate, current, or authorized. A polished response can still cite an irrelevant passage or omit the document that changes the conclusion.
The most useful evaluation separates retrieval quality from answer quality. This guide shows how to build a repeatable test set, inspect failures, and compare changes without mistaking a few successful demonstrations for dependable performance.
1. Define the tasks your assistant should complete
Start with concrete user questions. An internal assistant might need to identify the current refund policy, explain a troubleshooting procedure, compare product plans, or locate the owner of an operational process. Each task has different evidence and accuracy requirements.
Specify what success means before testing. For a policy question, success may require the current approved document, a faithful summary, and a citation to the relevant section. For a procedural question, the answer may need prerequisites, ordered steps, and a warning about unsupported environments.
Record unacceptable outcomes too. These can include inventing a policy, using an archived source as current, exposing restricted content, or producing instructions when the available evidence is insufficient. Evaluation becomes much clearer when the team agrees on these failure definitions.
2. Build a representative question set
Collect real questions where possible and remove personal or confidential details from evaluation material. Include common requests, ambiguous wording, abbreviations, multi-part questions, and queries whose correct answer is that the information is unavailable. Do not build the entire set from neatly phrased questions written by the same model you are evaluating.
For each question, record the expected evidence and a reference answer or answer rubric. A reference answer is guidance for assessment, not a requirement that every acceptable response use identical wording. Include the source version and access context so future document changes do not silently invalidate the test.
Keep a separate held-out set for final comparisons. If you repeatedly tune prompts and retrieval against the same questions, you can optimize for those examples without improving broader performance. Refresh coverage when users introduce new tasks or terminology.
3. Evaluate retrieval before generation
Inspect which passages the retriever returns and in what order. Ask whether the evidence needed to answer the question is present, whether irrelevant material dominates, and whether document chunking has separated a critical warning from the instruction it qualifies.
Useful retrieval measures include whether a relevant result appears among the top results and how much of the returned material is relevant. These measures depend on the labels and task definition; there is no single score that establishes quality for every knowledge base.
Investigate failures by layer. The document may not have been ingested, the index may be stale, the query may use different terminology, or the chunk may lack context. Changing the generation prompt will not repair missing evidence that never reaches the model.
4. Check groundedness and answer completeness separately
A grounded answer makes claims supported by the retrieved evidence. A complete answer addresses the user’s actual request and includes important constraints. An answer can be grounded but incomplete if it quotes one accurate sentence while omitting a required exception.
Review factual claims individually for higher-risk tasks. Check whether the cited source really supports the stated conclusion, not just whether it mentions the same topic. Pay particular attention to numbers, eligibility rules, dates, product limits, and steps that could change a user’s account or system.
Do not assume a citation makes a claim true. A model can attach a genuine document link to a statement that the document does not support. Evaluate citation correctness and coverage alongside answer correctness, and preserve enough retrieval context to investigate disagreements.
5. Test access control and untrusted content
Apply access control during retrieval according to the user or service identity. A model instruction telling the assistant not to reveal secrets is not a substitute for preventing unauthorized documents from entering its context. Test the same question with users who have different permissions.
Include documents containing irrelevant instructions, misleading claims, and quoted text that resembles a command. Retrieved content should be treated as evidence, not as authority to change the assistant’s behavior or reveal other information. Our AI prompt injection guide explains this trust boundary.
Also test source lifecycle events. Remove a document, revoke access, replace an approved policy, and verify how quickly the retrieval system reflects the change. A system that answers correctly today can still expose stale information tomorrow if indexing and authorization updates lag behind the source.
6. Evaluate abstention and clarification
Some questions cannot be answered safely from the available documents. The assistant should be able to state the evidence gap, ask a relevant clarifying question, or direct the user to an approved owner. A confident guess is not a successful response merely because it sounds helpful.
Include queries with conflicting sources and ambiguous entities. Assess whether the assistant notices the conflict, distinguishes current from outdated material, and avoids silently selecting a convenient interpretation. Define the expected escalation path for sensitive or unresolved questions.
Balance abstention against usefulness. An assistant that refuses every difficult question is not reliable in practice. Track unnecessary refusals separately from unsupported answers, and inspect whether better retrieval or clearer scope would allow an evidence-based response.
7. Compare versions with cost and latency in view
Run candidate configurations on the same controlled test set. Change one meaningful variable at a time where practical: chunking, retrieval strategy, model, prompt, or reranking. Keep source snapshots and evaluation settings stable enough to interpret the comparison.
Record answer quality, retrieval quality, latency, and operating cost. A change that improves a small accuracy metric while doubling response time may not fit the intended workflow. Look at distributions and high-impact failures rather than reporting only an average score.
Use automated evaluators as aids, not unquestioned judges. A model-based grader can be inconsistent or share blind spots with the model being tested. Calibrate it against human review, especially for policy, security, financial, or other consequential answers.
A minimal evaluation record
- User question and intended task.
- User access context and source snapshot.
- Expected relevant passages or supporting documents.
- Retrieved passages and generated answer.
- Correctness, groundedness, completeness, and citation assessment.
- Latency, cost, failure category, and remediation owner.
Keep evaluation data protected and remove unnecessary sensitive content. Logs used to improve an assistant can themselves become an information-exposure risk.
Frequently asked questions
Does RAG eliminate hallucinations?
No. It provides external context, but retrieval can fail and generation can misinterpret or invent details. Evaluation and runtime safeguards remain necessary.
Should I trust one benchmark score?
No. A benchmark may not reflect your documents, users, permission model, or tasks. Use representative local tests and inspect the failures behind aggregate results.
Where can I learn the architecture basics?
Microsoft’s RAG and indexes overview explains retrieval, grounding, and security considerations. Use that architectural context alongside your own measured evaluation results.
