TECHNICAL NOTE / 5 MIN READ

Design experiments that can change your next decision

Choose a learning objective, a credible comparison, and an outcome that reflects accepted work. Preserve enough context to revise the recommendation.

VALERY · PRINCIPAL, EML SYSTEMS

Experiments
Engineering methods
Decision making

THE PRACTICAL ANSWER

What makes an experiment worth running?

Name a choice you are prepared to revise, a comparison that can address it, and an accepted outcome that matters. Set interpretation and revision rules before looking at results, then retain the conditions, failures, and uncertainty behind the recommendation.

When to use this

Use this before committing to an AI feature, repository change, retrieval intervention, or evaluation approach. When the mechanism is unclear, an exploratory observation may be the useful first step.

Use the checklist and record

Begin with a choice you are prepared to revise

“Does better context help?” is too broad to guide an investigation. “Should this team add a short repository orientation file for maintenance tasks?” identifies an actionable choice. It also exposes the context that matters: task type, repository, agent configuration, users, and the alternatives under consideration.

State the learning objective before selecting a method. Observation may reveal which files an agent actually uses. A case study can explain how documentation and review interact. A controlled comparison can examine whether a particular intervention changes outcomes. These methods answer related but different questions.

Write an observable expectation

A useful hypothesis connects an intervention to a defined outcome. For example: a minimal orientation file reduces human corrections per accepted maintenance change without a consequential reduction in completion quality. Define a correction, an accepted change, and a consequential quality loss before interpreting the results.

When the mechanism is unclear, start with an exploratory aim instead of inventing precision. Inspect sessions and ask where project knowledge is missing. The result may suggest a better intervention, such as a validation script or a module-level contract, rather than a longer instruction file.

text

Decision: keep a minimal repository orientation file?
Context: maintenance tasks in one stable repository snapshot
Comparison: current setup versus the same setup plus orientation
Primary outcome: human corrections before acceptance
Guardrails: accepted completion, defects, review time, execution cost
Prediction: fewer corrections without material quality loss
Revision rule: narrow or remove guidance if benefits do not survive review

Keep the comparison interpretable

Record the repository state, task wording, model and agent versions, available tools, instructions, and acceptance process. Keep these conditions comparable when the aim is to attribute a difference to the intervention. A simultaneous model upgrade and instruction rewrite makes that attribution difficult.

Use tasks that reflect the decision’s scope and repeat executions when variability matters. Keep unsuccessful runs and protocol deviations visible. An agent that abandons a difficult task should not disappear from the analysis because it produced no final patch.

Choose tasks before knowing which configuration succeeds.
Use the same acceptance criteria across conditions.
Record interventions and configuration changes during a run.
Keep exploration examples separate from later challenge cases where practical.

Measure accepted work and human effort

A shorter agent run can still require more preparation, review, and repair. Measure the outcome the team needs, such as a correct change acceptable for merge. Include human steering, unintended edits, missed escalation, execution cost, and maintenance burden as guardrails or secondary outcomes.

Select a primary outcome so the recommendation cannot quietly switch to whichever metric improves. Capture qualitative observations as well. Two runs can have similar completion scores while one adds an unnecessary abstraction and the other exposes a useful missing contract.

Interpret the result at its actual scope

Ask how large the observed difference is, how variable it is, and whether it matters after accounting for cost and risk. Examine whether task difficulty, evaluator disagreement, changed tools, or a small number of unusual runs could explain the finding. Preserve examples that challenge the preferred interpretation.

A successful local comparison supports a local recommendation under the examined conditions. It does not establish a universal rule for every repository or agent. An inconclusive result may indicate that the effect is small, the measurement is weak, or the evidence is insufficient. Record what remains unresolved.

Connect the finding to the next commitment

Decide whether to retain the intervention, narrow its scope, revise it, gather another kind of evidence, or stop investing. Document the reason and the conditions that would prompt reconsideration. Useful negative results can prevent a team from maintaining guidance that does not earn its cost.

Keep the experiment record proportional to the decision. A small reversible workflow change may need a short comparison and a few trace examples. A consequential change to production authority requires stronger evidence and controls. The experiment is successful when it improves the decision, including a decision to change direction.

PUT THIS NOTE TO WORK

Write a decision-focused experiment record

Keep the record short enough to use during the work. Its purpose is to prevent the decision, outcome, or comparison from shifting unnoticed.

State the decision, owner, alternatives, and the observation that could change direction.
Choose a learning objective and a method that can answer it; separate exploration from claims about an intervention’s effect.
Define a primary outcome, acceptance criteria, guardrails, and comparable task and configuration conditions.
Retain unsuccessful runs, interventions, deviations, and observations that challenge the preferred explanation.
Assess effect, variability, relevance, cost, and evidence limits before choosing to retain, modify, investigate further, or stop.

Open the copyable working record

Copy this blank record into your project and fill it with your own evidence. Keep links to the underlying work.

# Decision-focused experiment record

Decision, owner, and timing:
Alternatives under consideration:
Learning objective or hypothesis:
Finding that would change the decision:

## Design before execution
Task population and selected cases:
Baseline and proposed intervention:
Relevant system, model, data, and tool versions:
Comparison method and conditions held stable:
Primary outcome and acceptance definition:
Guardrails, operating cost, and human effort:
Planned analysis and local revision rule:

## Evidence and recommendation
Runs, failures, deviations, and evidence locations:
Observed difference, variability, and competing explanations:
Scope limits and unresolved questions:
Decision: retain / modify / investigate / stop
Reason, owner, and conditions for reconsideration:

What you should leave with

Leave with a recommendation at the scope of the evidence, including an explicit account of what would make the team reconsider it.

A useful local comparison supports a local decision under the examined conditions. It does not establish a universal rule or justify expanding production authority without appropriate controls.

Adapted from our working notes on AI products and engineering. Explore the broader body of work at valery.tech ↗.

Back to technical notes