Choose a document collection with clear ownership
Start with one domain, such as onboarding guidance or internal purchasing procedures. For each document, record its title, owner, version, effective date, and permitted readers. Separate active documents, archives, and drafts. Ask the process owner to resolve conflicting rules first. A search system has no authority to decide company policy when two sources disagree, regardless of how confidently it phrases an answer.
Check whether the content can be read correctly
Scanned PDFs, complex tables, and process diagrams need extraction checks. Compare extracted text with the original page, especially numbers, footnotes, and exceptions. Preserve the relationship between text segments and their document sections. Use a sample that represents the range of formats instead of selecting only clean files. Microsoft’s RAG design guidance also emphasizes representative test material and evaluation of retrieval and answers.
Enforce access before information reaches an answer
The documents a user can retrieve should follow their permissions. Design restrictions into retrieval and test with different accounts. Hiding a document link from the interface does not establish whether its contents can appear in an answer. Discuss processing location, retention, and logging with the responsible teams for the project. Treat these as requirements to verify, with recorded decisions and owners.
Prepare test questions and reference answers
Ask users for realistic questions and have procedure owners identify the source passages that should answer them. Include ambiguous questions, unavailable information, and superseded rules. Assess whether the reference is appropriate and whether the answer is supported by it. An apparently credible link is insufficient when the cited paragraph does not support the conclusion. Record errors by type so improvements target the actual problem.
Design behavior for insufficient evidence
Decide when the system asks for clarification, reports that information was not found, or refers the user to a process owner. Specify human review for consequential actions involving staff, customers, or transactions. The interface should let users open sources and report problems. Content owners can then distinguish missing documentation from retrieval or summarization errors and choose the appropriate correction.
Make updates part of operations
Agree who approves new documents, retires old versions, and checks results after an update. Repeat evaluation when sources or system components change. KODE can discuss the scope of a document assistant, its initial sources, and acceptance criteria. For an introductory conversation, describe the document types and example questions without sending confidential contents. A focused collection is sufficient to begin a useful scoping discussion.