Retail AI Evaluation: Build a Test Set for Procedure Answers
Build reusable test cases that check whether AI procedure answers preserve facts, conditions, references and escalation boundaries.
A procedure-answer assistant can sound helpful while skipping the condition that changes what a store manager should do. Testing only a few familiar questions can leave that weakness hidden. A useful test set checks whether the answer preserves the procedure's facts, conditions and limits.
The Generative AI for Store Operations guide covers choosing and supervising a workflow. This article focuses on one narrower job: building reusable cases for an assistant that answers questions from approved operating procedures.
The method below is a Retail Sense management recommendation, not a validated benchmark. NIST's voluntary AI Risk Management Framework, published in 2023, and its 2024 Generative AI Profile support contextual evaluation and risk management. They do not prescribe this retail test template or establish a universal passing score.
Start with an answerable reference
Choose a bounded procedure and have its responsible owner confirm the reference material before drafting expected answers. Record the document identifier, revision, effective status and relevant section. Use approved demonstration material or fictional records; keep sensitive employee, customer and business information out of an informal test collection.
Separate the question from the evidence. The question represents what a manager might ask. The reference identifies what the assistant can legitimately use. An expected answer should describe required facts and conditions, rather than demand one exact sentence.
Where the source is silent, the reference must preserve that silence. A reviewer cannot add a familiar local practice and then penalize the assistant for failing to reproduce it. Ask the procedure owner whether to amend the approved procedure or treat the issue as unresolved.
Build a case record someone else can review
Give every case a stable identifier. A spreadsheet or controlled document can work if its versions and access are managed. Include:
- Question and intent: the exact question, the task it represents and any relevant context.
- Reference: approved source identifier, revision, section and the facts available to the assistant.
- Expected facts and conditions: what must appear, what depends on a condition and what remains unknown.
- Unacceptable claims: invented deadlines, omitted escalation, unsupported choices or other material errors.
- Expected reference behavior: which source should support the answer and how the reviewer can locate it.
- Expected boundary: whether to answer, clarify, explain missing information or refer to the designated person.
- Test record: date, actual setup, answer, reviewer findings and linked issue identifiers.
Record enough setup information to repeat the test: the source packet, instructions and product or model version where available. Preserve relevant settings that the actual interface exposes. Do not assume that retrieval logs, model identifiers or technical traces are available.
Microsoft's documentation on retrieval-augmented generation evaluation distinguishes grounding, relevance and completeness concepts. Those distinctions help structure a review; its product-specific metrics or thresholds are not retail acceptance criteria.
Cover six kinds of questions
Build around the procedure's decision points, not just its headings. Include these six case types:
- Ordinary: all required information is present. Check that the answer gives the supported steps without adding requirements.
- Conditional: a stated condition changes the required action. Check that the answer recognizes and applies it.
- Missing information: a necessary fact is absent. Check that the assistant asks for it or follows the documented escalation.
- Conflicting information: two supplied details cannot both be used as presented. Check that the answer exposes the conflict instead of choosing silently.
- Ambiguous request: the wording supports more than one interpretation. Check that the assistant requests the clarification needed to proceed.
- Out-of-scope request: the question asks for something the procedure does not establish. Check that the answer explains the limit without inventing policy.
The appropriate number of cases depends on the procedure's complexity and consequences. Cover meaningful branches and exceptions; there is no universal sample size or percentage that proves readiness.
Work through a fictional display-material request
Consider this entirely fictional procedure:
- Submit a display-material request using Form R and the current campaign code.
- If the code is missing or two codes are provided, ask the merchandising coordinator to confirm the current code.
- The procedure defines no turnaround time.
A test question could be: “I have campaign codes C-ALPHA and C-BETA. Which should I put on Form R, and can the materials arrive by Friday?”
The expected reference facts are straightforward: Form R is the request form; two codes require confirmation from the merchandising coordinator; the source does not establish delivery by Friday. Different wording is acceptable if those facts and limits survive.
These are illustrative answer fragments, not results from an actual model test:
- “Use Form R.” This is supported, but incomplete. It misses the conflicting codes, required escalation and unanswered deadline.
- “Use C-ALPHA on Form R and expect the materials by Friday.” This invents both a choice and a delivery promise.
- “Use Form R once the merchandising coordinator confirms the current code. The procedure does not specify turnaround, so Friday cannot be confirmed from this source.” This preserves the required conditions and boundary.
Expand the example into separate cases: one confirmed code, no code, two codes, an unclear request for “the campaign kit,” and a request to guarantee delivery. Each case should identify its own expected behavior.
Review four dimensions separately
Do not reduce every answer to whether it “looks right.” Use four distinct reviews:
- Support: Is each factual claim backed by the supplied approved reference?
- Completeness: Are the required steps, conditions and exceptions present?
- Reference accuracy: Does a cited document or section exist, match the tested revision and support the associated claim? If citations are unavailable, record that limitation.
- Boundary handling: Does the answer clarify or escalate appropriately when evidence is missing, conflicting or outside scope?
An answer can pass support while failing completeness, as the short Form R answer demonstrates. A correct document title also cannot rescue an unsupported statement. Keep material failures visible rather than letting strong performance elsewhere conceal them in an average.
Diagnose with the evidence you actually have
Compare the answer with the case record before proposing a fix. Was the expected reference wrong? Was the procedure unclear? Did the assistant receive the relevant section? Did the answer omit a condition that was present?
Inspect retrieved passages or other traces only when the actual system provides authorized access to them. Without that evidence, label retrieval or generation explanations as hypotheses. An observed unsupported deadline is enough to record a failure; it does not prove which internal component caused it.
AWS's operational guidance discusses validated reference answers, versioned evaluation and model-based assessment. Automated graders can help organize comparisons, but they can also be wrong. Have a responsible human adjudicate material findings against the reference, especially where graders and reviewers disagree.
Register failures and retest changes
Maintain a material-failure register with the case identifier, observed problem, operating consequence, evidence, responsible owner, proposed correction and retest status. Distinguish a source correction from an instruction change or a technical change so the team knows what was addressed.
Keep the original result. Create a new test-run version after a change and rerun the affected cases plus relevant neighboring cases. A fix for conflicting codes must not remove the ordinary request steps. Add newly discovered cases to a documented revision of the set.
A bounded decision can be to continue supervised use for the tested scope, revise and retest, or stop that workflow and return to the approved manual process. Record the scope, unresolved failures, reviewer and fallback. A passing test set does not justify broader automation or establish compliance, reliability or accuracy guarantees.
The practical result is a repeatable conversation about evidence: what the procedure says, what the answer preserves, what failed and what changed. Keep that record current when procedures or system behavior change.