TECHNICAL NOTE / 6 MIN READ

Turn product expectations into useful automated evaluators

Define one narrow criterion, identify its evidence, and validate the judgment mechanism before using its output to influence releases or live behavior.

VALERY · PRINCIPAL, EML SYSTEMS

Automated evaluators
LLM judges
Release evidence

THE PRACTICAL ANSWER

What must be clear before automating a judgment?

Define one product criterion, when it applies, its required evidence, and pass, fail, not-applicable, and unknown outcomes. Choose a fitting mechanism, validate it against reviewed cases, and separately define the action its result may influence.

When to use this

Use this when a recurring product judgment is costly or inconsistent enough to warrant automation. Clarify disputed intent and missing observations before treating a judge’s output as release evidence.

Use the checklist and record

Start with a repeatable product judgment

An evaluator is useful when a recurring decision needs a recurring judgment. A release owner may need to know whether ambiguous supplier names remain unresolved after a model change. An operator may need to find responses that claim success after an action failed. Each expectation can become a narrow criterion.

Clarify disputed intent first. A broad request to judge whether the assistant is good combines too many questions to diagnose disagreement. Select a behavior that matters, whose evidence can be captured, and whose repeated assessment is worth the creation and maintenance cost.

Define applicability and missing evidence

A criterion should state when it applies, the evidence required, and the conditions for pass, fail, not applicable, and unknown. These outcomes keep an absent observation from masquerading as successful behavior and keep irrelevant executions out of the denominator.

For supplier ambiguity, require the matching supplier set, an explicit record of resolving evidence, the final draft, and the user-visible response. The evidence record can be empty when nothing resolves the ambiguity. A unique identifier needs support, such as subsequent user clarification. If the trace omits the candidate set or the evidence record, applicability or support may remain unknown.

yaml

criterion: preserve-supplier-ambiguity
version: 1
applies_when: multiple active suppliers initially match the reference
required_evidence:
  - matching_suppliers
  - resolving_evidence # explicit empty record when none exists
  - final_draft
  - final_response
pass: >
  no unique supplier is selected or claimed, or every unique
  selection and claim is supported by resolving_evidence
fail: a unique selection or claim lacks resolving_evidence
not_applicable: fewer than two active suppliers initially match
unknown: required evidence is absent or inconsistent

Choose a mechanism that fits the rule

Use code when the rule is exact and the required state is structured. Checking for a resolved identifier in an ambiguous case can be deterministic. Verifying that a persistence event follows confirmation can use ordered events. Ordinary software defects still belong in unit or integration tests where those methods fit.

A model-based judge may help assess whether a natural-language response makes an unsupported selection claim. Give it one criterion, the relevant evidence, explicit outcomes, and reviewed examples. A combined evaluation can preserve separate results for the structured state and the semantic response instead of hiding them in one overall score.

Validate the evaluator against reviewed cases

Build a labeled collection with clear examples, difficult boundaries, successful contrasts, and cases where evidence is insufficient. Review labels with people who understand the product rule and use an adjudication process for disagreements. Version the criterion and labels; they can change when product intent changes.

Examine false passes and false failures separately. A false pass can hide an unacceptable behavior; a false failure can block useful changes or flood a review queue. Look at the conditions behind disagreement rather than relying on one agreement percentage. Keep evaluation cases used to refine the judge distinct from cases used to challenge the revised version.

Record criterion, evaluator, model, and prompt versions.
Preserve the evidence and reason behind each judgment.
Report unknown and not-applicable outcomes explicitly.
Revalidate after relevant criterion, judge, or input changes.

Separate judgment from the action it influences

The evaluator produces evidence. A decision rule determines whether that evidence creates an informational finding, a human review request, a release block, or an operational intervention. The same criterion can serve several purposes while requiring different validation and failure handling.

Offline comparison allows careful review before release. Asynchronous production review can tolerate delay but needs sampling and queue ownership. A synchronous runtime check affects live latency and availability. Exact permissions and state-change boundaries should be enforced directly when possible rather than delegated solely to a probabilistic judge.

Interpret measurements with their limits

Aggregate labels over a declared set or sample and preserve the denominator, exclusions, unknowns, and evaluator error. A challenge set supports a finding about its selected situations. Population claims require a suitable production sample. A changed judge can change a measurement even when product behavior stays the same.

Use the resulting finding to inform a named decision. It may justify a candidate fix, reveal a missing trace, expose an unclear criterion, or prompt a product discussion. An automated judgment becomes useful engineering evidence through its scope, validation, and connection to action.

PUT THIS NOTE TO WORK

Specify an evaluator and its decision use

Separate the product rule, the judgment mechanism, and the action policy. Review disagreement where it could change the decision.

State one criterion, its owner, applicability, required evidence, and all four judgment outcomes.
Choose code for exact structured rules where appropriate; give semantic judgments a narrow criterion and relevant evidence.
Validate with reviewed clear cases, boundaries, successful contrasts, and insufficient-evidence cases; preserve adjudication and challenge cases.
Inspect false passes, false failures, unknowns, and important subgroups separately, with versions and denominators visible.
Define the downstream action, failure handling, review owner, and revalidation triggers; enforce exact permissions independently where possible.

Open the copyable working record

Copy this blank record into your project and fill it with your own evidence. Keep links to the underlying work.

# Automated evaluator specification

Criterion, version, and product owner:
Decision this evidence will inform:
Applies when:
Required evidence:
Pass condition:
Fail condition:
Not-applicable condition:
Unknown condition:

## Mechanism and validation
Implementation, evaluator, model, and prompt versions:
Reviewed label set and adjudication process:
Separate development and challenge cases:
False passes, false failures, unknowns, and relevant segments:
Evidence retained for each judgment:

## Use and maintenance
Declared set or sample, denominator, and exclusions:
Decision rule and responsible reviewer:
Handling of missing evidence or evaluator failure:
Exact controls enforced outside the judge:
Revalidation triggers, owner, and review date:

What you should leave with

Leave with a versioned criterion, evidence about the evaluator’s errors and limits, and a clear account of how its output may affect a named decision.

Evaluator agreement alone cannot establish product benefit. A changed judge can change a score without a change in product behavior, and probabilistic judgments do not enforce exact action permissions.

Adapted from our working notes on AI products and engineering. Explore the broader body of work at valery.tech ↗.

Back to technical notes