How to build a RAG evaluation dataset from your company documents

An internal AI assistant should not be judged by five improvised questions in a demo. If the questions change after every answer, the team cannot tell whether retrieval improved, a prompt merely became more persuasive, or a restricted document influenced the result. A small, versioned RAG evaluation dataset turns those impressions into repeatable evidence.

This guide is for operations, IT, and knowledge owners. It shows how to create a practical test set from synthetic or approved company-style documents, record the expected evidence, include failure cases, and rerun the same questions after a source or configuration changes. It complements the 15-question internal AI search buyer checklist: that page helps compare tools, while this one builds the evaluation asset used in the comparison.

What belongs in a RAG evaluation dataset?

A useful record contains more than a question and a model answer. At minimum, capture:

• a stable case ID;
• the user persona or role;
• the question in natural language;
• the authoritative source expected to answer it;
• the required facts;
• facts or sources that must not appear;
• the expected outcome: answer, clarify, or decline;
• the access profile used for the test;
• the human owner who approved the ground truth;
• the dataset version and review date.

Microsoft's RAG evaluation guidance describes an evaluation set as curated queries, ideally with outputs, that represent the intended use and remain challenging, diverse, and current. Google Cloud recommends high-quality questions that cover the underlying data, wording variations, and realistic complexity, paired with a golden reference dataset. Amazon Bedrock's evaluation documentation makes the role of ground truth explicit: expected retrieved texts and expected responses allow the evaluator to compare the system with what should have happened.

“Golden” does not mean perfect forever. It means reviewed enough to be the current reference for a defined test. When a policy changes, the ground truth should change through a visible dataset revision.

Step 1: define the decisions the assistant must support

Do not begin by sampling random pages. List the recurring decisions or interruptions the assistant is meant to reduce: approving an expense, explaining a leave rule, finding a contract clause, preparing a proposal, troubleshooting a machine, or locating the current onboarding step.

Choose two or three personas and ask their real subject-matter owners for recent questions. Remove customer, employee, and confidential details unless you have explicit authorization to use them. A synthetic pack is often safer for vendor evaluation because every fact and permission can be controlled.

Start with 20 to 40 cases if that is enough to cover the first workflow. Coverage matters more than a large round number. A hundred near-identical questions can hide a missing permission or freshness test.

Step 2: freeze a small source pack

Select a controlled set of documents and record its version. Include the types your team actually uses: a policy, a procedure, a table, a short FAQ, and a longer manual. Give every source an owner and an approval date.

Add deliberate friction:

• a current policy and an obsolete draft;
• two documents with similar titles;
• one fact spread across two sources;
• one restricted document;
• one ambiguous question that needs clarification;
• one plausible question whose answer is absent.

The absent-answer case is essential. A fluent invention is not a partial success. The expected result is a clear limitation, followed by a useful next step. The grounding glossary explains why a related citation does not automatically support a claim.

Step 3: write questions before looking at the output

Write questions the way employees speak, not as copies of document headings. Include paraphrases, abbreviations, misspellings, exact identifiers, and multi-part requests. Keep the original wording even if it is untidy; normalization can erase the difficulty you meant to test.

For every question, identify the smallest authoritative passage that supports the answer. Then write required facts as short, independently checkable statements. Avoid a single polished reference paragraph that encourages superficial text matching.

A good record might say:

Step 4: cover twelve test classes

Use this matrix as a starting point. Replace the examples with approved questions from your own work.

1. Direct fact: one answer in one clearly named source.
2. Paraphrase: the question avoids the document's terminology.
3. Exact identifier: an invoice code, product reference, policy number, or error code.
4. Multi-source synthesis: required facts live in two compatible documents.
5. Current versus obsolete: a superseded draft looks more similar than the approved source.
6. Permission allowed: the authorized role should receive the restricted answer and citation.
7. Permission denied: an unauthorized role must not see the answer, title, snippet, or citation.
8. Missing knowledge: no source contains the requested fact.
9. Ambiguous request: a safe answer requires a clarifying question.
10. Conflicting sources: two approved-looking files disagree and the conflict must be surfaced.
11. Boundary wording: a misspelling, acronym, or colloquial phrase tests retrieval robustness.
12. Unsupported instruction: a document contains text that should not redirect the assistant away from the user's task.

Keep security cases as hard gates. A permission leak should not be averaged away by high scores on easy questions.

Step 5: score retrieval and the answer separately

A system can retrieve the correct passage and still produce an incomplete answer. It can also write a plausible answer after retrieving the wrong passage. Record both layers.

For retrieval, check whether the expected source appeared, whether an obsolete or forbidden source appeared, and whether the cited passage contains the required evidence. For the answer, check required-fact coverage, unsupported claims, correct handling of missing information, usefulness, and clarity.

Do not turn the first rubric into a fake universal benchmark. Choose pass conditions for the business risk. A cafeteria FAQ and a contractual renewal decision do not deserve the same review threshold. NIST's AI Resource Center frames testing, evaluation, verification, and validation as lifecycle practices; the important habit is documenting the test and rerunning it when the system or use changes.

Step 6: add human review where judgment matters

Automated evaluators can help run a larger set consistently, but a domain owner should approve the reference facts and review nuanced failures. Google Cloud specifically recommends involving stakeholders and end users and combining quantitative evaluation with human judgment.

Use disagreement productively. If HR and operations cannot agree on the expected answer, the assistant has exposed a source-governance problem rather than an evaluation failure. Record it as a knowledge gap, assign an owner, and correct the authoritative document before changing prompts.

Step 7: version the dataset and change one variable

Store the source-pack version, dataset version, system configuration, date, and result for each run. When testing an improvement, change one meaningful variable at a time: the source, chunking strategy, retrieval method, prompt, or model. Google Cloud recommends holding the questions and reference answers steady so score changes can be attributed to the tested change.

Add new cases from production only after privacy review. Repeated unanswered questions and negative feedback are valuable candidates, but they should be cleaned, approved, and mapped to the correct source before entering the golden set. Retire cases when the business workflow disappears; do not silently rewrite their history.

A copyable CSV header

A spreadsheet is enough for a first version:

case_id,persona,question,expected_source,required_facts,forbidden_facts,expected_outcome,access_profile,owner,dataset_version,reviewed_at

Use one row per question. Put multiple required facts in a structured list or a consistent delimiter. Link source IDs to a frozen manifest rather than pasting confidential passages into an unprotected sheet.

Running the dataset with Polp

Polp's public help recommends starting with a small reviewed document pack, asking a real question, and checking whether the cited sources are correct. That is the right scope for a first evaluation run.

Upload or connect only the approved synthetic sources, then run every case with its specified role. Save the answer and citations, mark missing or unexpected evidence, and route unresolved source problems into the knowledge-quality workflow. The upload guide helps prepare clear, current files, and the asking guide explains how to inspect sources and treat a no-information answer as useful rather than encouraging a guess.

If you want to test this dataset against a source-backed internal assistant, request a Polp demo and bring the source manifest, roles, and pass conditions. A disciplined pilot should leave you with a reusable quality asset, not just a memorable demo.

The decision rule

Your evaluation dataset is ready when another reviewer can run the same cases against the same source version and understand why each result passed or failed. Keep the questions realistic, the expected evidence explicit, the permission cases uncompromising, and the revision history visible. That is enough to replace “the answers looked good” with a test your team can repeat.