TECHNICAL NOTE / 5 MIN READ

Design evaluation coverage around the decision

Build cases that exercise meaningful product situations. Make the relationship between the decision, the coverage, and the claims explicit.

VALERY · PRINCIPAL, EML SYSTEMS

Evaluation design
Test cases
Product behavior

THE PRACTICAL ANSWER

What does useful coverage require?

Start with the decision and the behavior it depends on. Choose realistic cases that exercise supported situations, important boundaries, and known failures. Preserve the fixture, expected condition, and judgment evidence for each case, then state which conclusions the selection can support.

When to use this

Use this when planning a release comparison or revising a case set whose purpose is unclear. Coverage review can expose untested situations; estimating their live frequency requires an appropriate population sample.

Use the checklist and record

Write the question before collecting examples

Suppose a purchase-request assistant proposes a supplier, amount, and approving manager. The team is deciding whether a revised matching step is suitable for a limited release. An answer requires more than a pile of example messages. It requires cases that distinguish the behavior the release promises to support.

A useful question is: does the assistant resolve a supplier when the available records establish one match, and preserve uncertainty when they do not? This question identifies a behavior and a boundary. Whether employees complete requests faster is a separate question requiring observation of the complete workflow.

Specify situations as well as inputs

The message “Order two monitors from Acme” can be clear in one environment and ambiguous in another. One supplier record may match Acme, or several active records may share that name. Evaluating only the message discards the state that determines which response is appropriate.

Represent a case with an input, the minimum fixture required to establish the situation, and an expected condition. Keep the condition separate from the judgment rule. “Two suppliers match” describes the situation. “Do not select a supplier without clarification” states the behavior to evaluate.

yaml

id: supplier-ambiguous-01
input: Order two monitors from Acme
fixture:
  suppliers:
    - { id: acme-office, name: Acme, active: true }
    - { id: acme-industrial, name: Acme, active: true }
expected_condition:
  supplier_reference: ambiguous
coverage: [supplier-ambiguity]

Choose dimensions that change the behavior

Derive coverage requirements from supported workflows, product boundaries, known failures, and important uncertainty. For supplier matching, useful dimensions may include unique, ambiguous, inactive, and absent matches. A missing delivery location may matter when it changes the supplier options. Decorative wording differences may add little.

Select combinations deliberately. Include ordinary successful requests, important boundaries, known regressions, and interactions with plausible consequences. Generating every combination can spend review effort on unrealistic cases while leaving the meaningful boundaries poorly understood.

Map each case to the coverage requirement it exercises.
Include similar successful cases to test the boundary of a failure.
Record exclusions that could matter to the release decision.

Review constructed cases for realism

Real inputs reveal natural omissions, shorthand, and assumptions. Preserve their provenance and apply the appropriate privacy treatment. Written or generated inputs can fill a declared gap, but they need review: does the wording resemble actual work, and does the fixture create the intended condition?

Generate the user input independently from the expected answer. Otherwise the example may encode the judgment into its wording. Remove near-duplicates and cases that announce the hidden condition, such as a request explicitly telling the assistant that the supplier name is ambiguous.

Separate coverage from population measurement

A challenge set can deliberately concentrate difficult cases. Its failure rate describes that set. It cannot establish how often customers encounter those failures. A production estimate needs a declared population, time window, selection method, and handling of missing observations.

Keep diagnostic, regression, comparison, and production-sampling purposes visible. They can share cases, but their results support different conclusions. Coverage tells you which situations have been examined; the sampling method determines whether their prevalence says anything about live use.

Treat the first set as a maintained asset

Before execution, review whether every material requirement has a realistic case and a fixture that actually activates it. Confirm the relevant time, state, data, and system versions can be recorded. Decide how missing evidence will be reported.

Expand the set when production, failure analysis, or a new decision exposes a gap. There is no useful universal case count. Start with a reviewable set that answers the stated question, preserve its limits, and let new evidence earn the next addition.

PUT THIS NOTE TO WORK

Build a coverage decision record

Use these checks before adding more examples. The output should connect each material claim to a case and the evidence needed to assess it.

Name the decision, owner, supported scope, and behavior the evidence must examine.
List situations that change the expected behavior. Include ordinary work, consequential boundaries, regressions, and useful successful contrasts.
Check that each fixture actually creates its stated situation and that the expected condition is separate from the judgment rule.
Record how cases were selected, which requirement each covers, and what was excluded.
Decide how missing observations, unknown outcomes, and reviewer disagreement will affect the recommendation.

Open the copyable working record

Copy this blank record into your project and fill it with your own evidence. Keep links to the underlying work.

# Evaluation coverage decision record

Decision and owner:
Supported users, tasks, and conditions:
Behavioral claim under review:
Case-set purpose: diagnosis / regression / comparison / population estimate

## Coverage row — repeat for each material situation
Situation and why it matters:
Input and state fixture:
Expected condition:
Judgment rule and required evidence:
Case IDs and source or construction method:
Successful contrast or boundary case:

## Scope and next decision
Selection method and relevant population:
Exclusions and missing observations:
Unknown outcomes and how they will be handled:
What result would change the recommendation:
Case, system, and data versions:
Recommendation, uncertainty, owner, and review date:

What you should leave with

Leave with an inspectable map from a decision to claims, situations, cases, and evidence, plus the gaps that could still change the recommendation.

This record organizes coverage. A completed checklist or a large case count does not establish production prevalence or justify a release by itself.

Adapted from our working notes on AI products and engineering. Explore the broader body of work at valery.tech ↗.

Back to technical notes