AI Feature Test Plan Worksheet

Use this worksheet before shipping an AI-assisted feature or expanding an existing one. It is meant for builders who need a practical test plan, not a research benchmark.

The goal is to decide whether the feature behaves well enough for its intended use. Keep the plan small, concrete, and tied to release decisions.

Feature Snapshot

Fill this in before writing test cases.

• Feature name:
• User or workflow served:
• AI role: summarize, classify, draft, extract, recommend, route, or other:
• Source material the model may use:
• Tools or actions the feature may call:
• Tools or actions the feature may not call:
• Human review point:
• Production impact if wrong:
• What does AI decide versus only suggest?
• Blast radius if the output is wrong:

Risk tier:

• low: output is advisory and easy to reverse
• medium: output affects a workflow, queue, or human decision
• high: output could trigger external action or material user harm

Scope Boundary

Write the boundary in plain language.

The feature should:

• [Describe the approved behavior.]

The feature should not:

• [Describe behavior that is outside the boundary.]
• [Describe cases this test plan is not covering.]

The feature must escalate or refuse when:

• [Describe stop conditions.]

Expected Behavior First

Write expected behavior before running the model. If you decide what “good” means after seeing a fluent answer, the test is already biased.

For each case, define:

• required source facts or fields
• allowed variation in wording
• unacceptable additions
• required refusal or escalation behavior
• acceptable uncertainty labels

Example target:

The feature may draft a summary, but it must not invent an owner, date, cause, or release recommendation. Missing details must be labeled.

Test Set Mix

Start with 10 to 15 cases. Add more after the first production learning cycle.

Suggested mix:

• 4 golden path cases
• 3 edge cases
• 2 blocked or escalation cases
• 1 to 3 regression cases from known failures
• 1 adversarial or instruction-conflict case when the feature accepts free text
• 1 empty, malformed, or low-context case when the input can be incomplete

Do not skip blocked cases. A feature that answers when it should stop is often more dangerous than a feature that fails loudly.

Keep the set small enough to rerun after every prompt or model change.

Test Case Template

Copy this block for each case.

Case ID:
Case type: golden path | edge | blocked | regression
Input:
Source material:
Expected behavior written before run:
Expected behavior:
Must include:
Must avoid:
Pass criteria:
Failure severity: block release | require review | log for follow-up
Reviewer notes:

Scenario Design

Use a mix that matches how the feature can fail:

Include normal work and safe-failure cases.

Gate Definitions

Define gates before reviewing output.

Example targets, adjust by risk tier:

• golden path: at least 90 percent pass
• blocked / escalation: 0 missed refusals or escalations
• high-impact regression: 0 returns
• sensitive or external-action cases: 0 unsafe actions prepared without review

Block Release

The feature cannot ship if any of these occur:

• unsupported facts, citations, numbers, dates, or owners
• required refusal or escalation is missed
• sensitive data is exposed or mishandled
• the output changes the meaning of source material
• an external or high-impact action is prepared without approval
• a known regression returns

Project-specific blockers:

• [Add blockers.]

Require Review

The feature needs review before release if:

• output is mostly correct but wording could mislead the user
• an edge case is handled inconsistently
• required structure is partially missing
• the prompt or source packet needs clarification
• reviewer confidence is low but no blocker occurred

Project-specific review triggers:

• [Add review triggers.]

Log For Follow-Up

The feature can ship while tracking:

• minor formatting issues
• optional detail omissions
• low-risk tone improvements
• duplicate or awkward wording

Project-specific follow-up items:

• [Add follow-up items.]

Results Table

Use one row per case.

Regression Habit

After every prompt, model, retrieval, tool, or policy change:

• re-run the test set
• add any serious failure as a regression case
• record the model and prompt version
• record the source packet or retrieval configuration
• compare results against the last accepted run

Also rerun after:

• model parameter changes
• tool permission changes
• dependency or retrieval-index changes
• new input format or UI flow changes
• production incident, near miss, or reviewer override

If the feature depends on retrieval or tools, test at least one case where the required source or tool is unavailable. The expected result should be an honest stop, partial answer, or escalation.

Worked Mini-Example

Feature: draft a short triage summary from a support note.

Release decision:

Ship only if Case B does not invent a cause and Case C escalates. If either fails, hold release even if Case A looks good.

Release Decision

Choose one:

• Ship: all blockers passed, review items accepted, follow-up logged.
• Hold: one or more blockers remain.
• Limited pilot: blockers passed, but usage stays narrow until review items are resolved.
• Redesign: test results show the feature boundary or workflow is wrong.
• Ship with guardrail: blockers passed, but release requires added review, narrower audience, or reduced tool permissions.

Decision:

Gate evidence:

Known limits:

Release owner:

Reviewer:

Date:

Reference Links

These public references are useful starting points for deeper study. They are linked for attribution and further reading; the worksheet above is synthesized as original LIW training guidance.

• OpenAI Evals - useful for learning how repeatable examples and graders support model evaluation.
• NIST AI Risk Management Framework - useful for connecting evaluation evidence to risk decisions.
• OWASP Top 10 for Large Language Model Applications - useful for identifying unsafe output and adversarial-input scenarios to include in test sets.