Test the awkward questions before your customers do.

Write expected behaviour before reading the answer.

For each test, record the customer question, the relevant approved source and what a useful response must include. Also record what it must not claim. Some cases should require clarification or escalation rather than a complete answer. A reviewer needs these expectations to judge the result consistently.

Include ordinary variations in wording, short messages and follow-up questions. If the assistant serves several languages, evaluate the actual questions and source coverage in each language. Do not use the English result as evidence that the other languages are ready.

Test more than factual recall.

Try conflicting or outdated source scenarios in a controlled environment. Ask about services that are not offered and request commitments the assistant cannot make. Include attempts to override its instructions or obtain restricted information. The objective is to identify failures and narrow the scope where necessary.

• Supported question: an accurate, useful answer from approved information.
• Ambiguous request: a relevant clarification rather than a guess.
• Missing information: an honest limit and a useful next step.
• Restricted or out-of-scope request: the defined boundary holds.
• Handoff request: the customer reaches the agreed receiving process.

Verify the receiving side of the conversation.

Complete controlled handoffs and inspect what the team receives. Check the destination, context, language and any consent or contact fields required by the agreed process. Simulate a delivery failure and confirm that the interface does not report success. Use synthetic data and clearly mark test enquiries.

Review the customer experience when a person is unavailable. The assistant should describe the next step accurately. It should not promise an immediate response or completed action unless the operating process supports that promise.

Set release criteria and keep the test set.

Classify failures by their consequence and agree which ones block launch. Correct the source, conversation design or integration, then rerun the relevant cases and nearby variations. Preserve the test set so later content and platform changes can be checked against the same expectations.

After launch, review unresolved questions and staff feedback with appropriate access controls. Add representative new cases to the evaluation set. Improvement should be tied to useful answers and reliable handoffs, not simply longer conversations or a larger number of automated replies.