An AI-assisted feature can pass normal software tests and still fail the user. The route can render, the API can return a response, and the database write can succeed, while the model quietly gives an unsupported answer, ignores an instruction, refuses a valid request, or handles edge cases inconsistently.
That does not mean AI features are impossible to test. It means builders need a second layer of testing: examples that represent the work, expected behavior that can be reviewed, and release gates that say when the feature is good enough to use.
Why Ordinary Tests Are Not Enough
Traditional tests are strongest when behavior is deterministic. Given the same input, the function should return the same output. That still matters for AI-assisted systems. Authentication, routing, permissions, storage, logging, and UI behavior should remain ordinary software with ordinary tests.
The model-facing part is different. A model can produce more than one acceptable response, and small changes to prompts, context, tools, or model versions can change output quality. A brittle test that expects one exact sentence is usually the wrong tool.
Instead, test the behavior that matters:
- Did the answer use the supplied source material?
- Did it avoid unsupported claims?
- Did it refuse or escalate when the request was outside scope?
- Did it preserve required facts, names, dates, and numbers?
- Did it produce the expected structure?
- Did it stay inside privacy, security, and audience rules?
- Did it tell the user what it could not verify?
The goal is not to remove judgment. The goal is to make judgment repeatable.
Start With A Small Test Set
A test set is a collection of representative examples. Each example contains the input, the expected behavior, the review criteria, and the release decision if the example fails.
Start small. Ten carefully chosen examples are more useful than a large pile of unclear examples.
Include four kinds of cases:
Golden Path Cases
These are normal requests the feature should handle well.
Example: a user submits a short status note and asks the system to summarize blockers, decisions, and follow-up actions.
Expected behavior:
- summarize only what appears in the note
- preserve the owner and due date when present
- label missing owners or dates instead of inventing them
- return the required sections
Edge Cases
These are valid requests that stress the boundaries.
Examples:
- the source text is long and repetitive
- the user includes conflicting dates
- the input has incomplete context
- the task asks for a short answer but the source has many caveats
Expected behavior: the feature should handle ambiguity honestly and preserve important uncertainty.
Blocked Or Escalated Cases
These are requests the feature should not answer directly.
Examples:
- the user asks the system to decide something outside its authority
- the request requires missing source material
- the prompt includes sensitive information the feature is not approved to process
- the output would trigger an external action without review
Expected behavior: refuse, escalate, or return a draft-only answer depending on the product rule.
Regression Cases
These are examples that failed before or represent known risk.
Keep them after each bug fix, prompt change, model change, or tool change. A regression example is a scar with a purpose: it keeps the same mistake from returning quietly.
Define Expected Behavior Before You Run It
Do not run the model first and then decide what “good” means. That invites confirmation bias.
For each test case, write:
- Input: the user request and any source material.
- Expected behavior: what the feature should do.
- Must include: required facts, sections, citations, labels, or warnings.
- Must avoid: unsupported claims, forbidden actions, private data exposure, or wrong tone.
- Failure severity: blocker, review-needed, or follow-up.
If the expected behavior cannot be described, the feature is probably not ready for automation.
Use Pass/Fail Gates
A useful evaluation gate connects test results to action. It does not merely produce a score.
Use a simple three-tier gate:
Block Release
Block release when the feature:
- invents facts, citations, numbers, or owners
- ignores a required refusal or escalation rule
- exposes or mishandles sensitive data
- changes the meaning of source material
- takes or prepares an action outside the approved boundary
- fails a high-impact regression case
Require Review
Require review when the feature:
- gives a mostly correct answer with weak wording
- omits useful but nonessential context
- handles an edge case awkwardly
- needs a clearer prompt, rubric, or source packet
- passes technically but leaves reviewer confidence low
Log For Follow-Up
Log for follow-up when the issue is low risk and does not block use:
- minor formatting mismatch
- small tone adjustment
- wording preference
- missing optional detail
- duplicate sentence or awkward transition
This tier matters. If everything blocks release, the team stops trusting the gate. If nothing blocks release, the gate is theater.
Re-Run The Set When The System Changes
AI behavior can shift when any part of the system changes.
Re-run the test set after:
- prompt changes
- model or provider changes
- retrieval or source-selection changes
- tool permission changes
- new data formats
- UI flow changes that alter user input
- policy or compliance rule changes
Record the exact prompt version, model, tool settings, source packet, and date. Without versioning, you cannot tell whether a later failure is new or old.
Worked Example: Support Note Summarizer
Imagine a feature that turns internal support notes into a short triage summary. The feature is not allowed to invent missing facts or assign priority without evidence.
Test case:
- Input: “User cannot export the monthly report. Error appears after clicking Download. No workaround confirmed. Alex is checking logs.”
- Expected output: concise summary, known owner, unresolved status, no invented root cause.
- Must include: export problem, monthly report, error after Download, Alex checking logs.
- Must avoid: claiming a fix exists, assigning severity without evidence, naming a due date.
- Failure severity: blocker if it invents a fix; review-needed if it omits the owner.
Good output:
Export of the monthly report is failing after the user clicks Download. Alex is checking logs. No workaround or root cause is confirmed yet.
Bad output:
The monthly report export is failing because of a logging error. Alex will fix it today.
The bad output sounds helpful, but it invents cause and timing. That is a release-blocking failure for this feature.
Builder Checklist
Before production use, confirm:
- the feature has at least one small labeled test set
- expected behavior is written before model output is reviewed
- blocked, review-needed, and follow-up results are separated
- regression cases stay in the test set after fixes
- model, prompt, source, and tool versions are recorded
- someone owns review and release decisions
- the system fails closed for high-impact or sensitive actions
If the team cannot answer these questions yet, the next step is not more autonomy. The next step is a clearer evaluation loop.
Reference Links
These public references are useful starting points for deeper study. They are linked for attribution and further reading; the lesson above is synthesized as original LIW training guidance.
- OpenAI Evals - useful for understanding test sets, graders, and repeatable model evaluation.
- NIST AI Risk Management Framework - governance reference for risk measurement, monitoring, and oversight.
- OWASP Top 10 for Large Language Model Applications - useful for thinking about adversarial prompts, data exposure, and unsafe output behavior.