Reliable AI answers: sources, permissions, and evaluation

Define what a good answer must contain

Specify the answer contract: a direct response, supporting sources, the applicable product or policy version, and a clear route when evidence is missing. The contract should also identify which questions require a live system lookup rather than a document search.

A public answer page and an internal assistant solve different problems. A public page should be readable and linkable on the web. An internal assistant must enforce access rules and protect private records. Publishing an internal knowledge base is not a shortcut to a safer assistant.

Design the evidence lifecycle

Maintain a source register with an owner, effective date, version, and access scope. Index a document only after those attributes are known. When it changes, replace or invalidate stale chunks and verify that deleted material no longer appears in results.

Retrieval systems can search for semantically related material, but related text is not automatically sufficient evidence. Product versions, regions, and customer entitlements often require explicit filters or an authorized record lookup. Keep those constraints in application logic.

Answer, qualify, or route

Use three visible outcomes. Answer when the evidence supports the requested claim. Qualify when it supports only part of the answer. Route to a person when documents conflict, permissions prevent access, or the question falls outside scope.

Illustrative response: “The current guide documents CSV export. I could not verify XLSX export for this product version. Here is the guide, and here is the team that can confirm the missing format.” This is more actionable than inventing a format or refusing the whole question.

Test the failure cases before launch

Build a small evaluation set with answerable questions, missing evidence, outdated policies, contradictory documents, and cross-account access attempts. Score the retrieved evidence separately from the final response. Record whether citations actually support the claim and whether the system correctly withheld inaccessible content.

Our RAG source contract guide covers ingestion and filtering. The answer evaluation guide supplies a worked test matrix. See the broader technology architecture for workflow boundaries.

Sources and editorial scope

Engineering guidance by Arrow AI, published and reviewed October 15, 2026. Worked examples are illustrative designs; they are not customer deployments or measured results.

• OpenAI: retrieval
• OpenAI: evaluation best practices