How to evaluate internal AI search tools: a 15-question checklist

An internal AI search demo can look convincing while avoiding the questions that matter after launch. A polished answer to a carefully chosen prompt does not show whether the system respects permissions, finds the latest policy, admits that evidence is missing, or remains useful when employees ask vague questions.

The practical way to compare tools is to test them against the same small, synthetic knowledge pack and score observable behaviour. This checklist gives operations, IT, security, and knowledge-management teams a repeatable method. It does not assume that one architecture or vendor is always best.

Start with a controlled evaluation pack

Create 15 to 25 fictional documents that resemble your real mix without using customer or employee data. Include a current policy, an obsolete version, a spreadsheet, a meeting note, two documents with similar titles, one restricted document, and one question whose answer is absent.

Write ten realistic questions before uploading anything. For each question, record the expected source, the facts that must appear, the facts that must not appear, and who should be allowed to see the answer. This prevents the team from changing the test after seeing a vendor’s output.

Microsoft’s current RAG architecture guidance separates retrieval, filtering, ranking, and reranking decisions. Google Cloud’s evaluation guidance likewise treats groundedness, instruction following, and answer quality as distinct dimensions. The lesson for buyers is simple: “it answered” is not a sufficient pass condition.

Retrieval and relevance

1. Does it find exact terminology?

Ask about an invoice number, policy code, product SKU, or unusual acronym. Exact identifiers expose systems that rely too heavily on broad semantic similarity. A strong tool should preserve precise keyword matching while still understanding natural language.

2. Does it understand paraphrases?

Ask the same question without using the document’s wording. This tests whether semantic search can bridge everyday language and formal company terminology.

3. Does it rank the authoritative document first?

Plant a current policy and an older draft with overlapping language. The system should prefer the approved, current source, not merely the most textually similar passage.

4. Can it explain which source supported which claim?

A list of documents is weaker than claim-level traceability. Check whether a reviewer can move from an answer to the exact passage that supports it.

5. What happens when the answer is missing?

The correct result may be “I cannot find that in the connected sources.” Reward calibrated refusal. Do not reward a fluent guess.

Permissions and confidentiality

6. Are source permissions enforced before retrieval?

Test the same question with two synthetic roles. A restricted document must not influence the unauthorized user’s answer, citations, snippets, or suggestions.

7. Can administrators explain access inheritance?

Ask how permissions from Drive, SharePoint, or another source are represented and refreshed. “The model was told not to reveal it” is not an access-control design.

8. How does it handle malicious instructions inside documents?

Include a harmless prompt-injection fixture telling the assistant to ignore its task. OWASP identifies prompt injection and sensitive-information disclosure as recurring application risks. The test is whether untrusted document text can redirect behaviour or expose unrelated context.

Freshness and lifecycle

9. How quickly does an update become searchable?

Change one synthetic policy and measure the interval until the new version appears. Record whether the old version remains visible and how conflicts are resolved.

10. Can deleted or revoked content disappear reliably?

Removal is part of correctness. Ask for evidence that deletion and permission revocation propagate through indexes, caches, and generated answers.

11. Can teams identify stale knowledge?

The tool should help owners see outdated, conflicting, or unanswered areas instead of hiding them behind a confident interface.

Answer quality and evaluation

12. Are groundedness and helpfulness measured separately?

An answer can be fully supported yet fail to answer the question. Google Cloud documents groundedness and question-answer quality as separate evaluation dimensions. Score both.

13. Can you replay the same test set after changes?

A useful evaluation process keeps questions, expected sources, permissions, and results versioned. NIST’s AI RMF resources emphasize testing, evaluation, verification, and validation across the lifecycle rather than a one-time demo.

14. Are latency and quality trade-offs visible?

Reranking can improve relevance but adds work after first-stage retrieval. Ask vendors to show response-time percentiles on your test pack, not only their fastest example.

15. Can reviewers inspect failures without reading every conversation?

Look for a workflow that surfaces unanswered questions, low-confidence results, missing citations, and recurring source gaps. A good operating model turns failures into an improvement queue.

Use a simple scorecard

Score each question from 0 to 2:

• 0: absent, unverifiable, or fails the fixture.
• 1: partially works but needs manual interpretation or has unclear controls.
• 2: passes with observable evidence.

Keep security gates separate from the total. A tool that leaks a restricted fixture should fail regardless of its aggregate score. For the remaining checks, compare the total, the weakest category, and the operational effort required to keep the system healthy.

Run the same evaluation with at least two roles and repeat it after changing a policy. That produces evidence about retrieval, permissions, and freshness instead of a collection of demo impressions.

What to ask in a real Polp evaluation

Polp’s public product page describes answers grounded in connected company documents with traceable sources. Treat that as a claim to verify, not a reason to skip testing. Use the checklist with a synthetic folder, inspect the returned sources, and test the same question under different permissions.

The AI governance toolkit can help assign owners and review responsibilities. If you want to run this scorecard against Polp, request a demo and bring your prepared fixtures and expected answers.

Decision rule

Choose the tool that performs best on your own controlled questions, respects every access boundary, and gives your team a practical way to improve failures. Do not choose solely on model name, the number of connectors, or a single impressive answer.

The most useful buying process is small, reproducible, and difficult to game: fixed fixtures, fixed roles, fixed questions, visible sources, and a written pass condition.