TECHNICAL NOTE / 5 MIN READ

Diagnose failures from the complete execution

Trace the first observable departure from expected behavior, follow its consequences, and preserve the questions the evidence cannot yet answer.

VALERY · PRINCIPAL, EML SYSTEMS

Failure analysis
Observability
Product behavior

THE PRACTICAL ANSWER

Where should a failure investigation begin?

Reconstruct the execution and establish the expectation before naming the failure. Locate the earliest divergence the retained evidence demonstrates, follow its consequences, and keep causal explanations as hypotheses until another observation or intervention distinguishes them.

When to use this

Use this for a recurring incident, a disputed failure, or a patch that keeps moving the symptom. When the trace or product expectation is missing, the first useful finding may be an evidence or intent gap.

Use the checklist and record

Reconstruct the job before naming the failure

A purchasing assistant says that an order request has been submitted. The user later discovers that nothing reached the approval queue. Calling this a hallucination records an impression, but leaves the engineering work unclear. Did submission fail, did the system never attempt it, or did the interface display an old result?

Collect the relevant sequence: the request, initial state, selected records, proposed action, confirmation, tool result, state changes, and final response. Include system and data versions. The trace should let another reviewer recover the behavior that supports the finding without relying on the original analyst’s memory.

Check that a verdict is possible

Two questions come before a product-failure label. Is there enough evidence to establish what happened? Is the applicable product expectation clear? A missing tool result creates an evidence gap. A disagreement about whether confirmation was required may reveal a product-intent gap.

Record these gaps explicitly. Treating an unobservable execution as a pass hides instrumentation problems. Treating every unexpected response as a failure lets reviewers invent requirements after seeing the result. A provisional observation can remain useful while the expectation or evidence is clarified.

Evidence gap: identify the missing event, state, or source needed to judge.
Intent gap: identify the unresolved product rule and its owner.
Confirmed failure: link the observed behavior to the expectation it violates.

Find the earliest observable divergence

Suppose the trace shows an accepted confirmation, a rejected submission, and a final message claiming success. The first demonstrated divergence may be the success claim after the rejected action. The rejected action itself may be an acceptable operational event if the product promises to report it and offer recovery.

Separate the observed condition, the divergence, its propagation, and the consequence. A wrong supplier selection might appear in a draft, survive approval, and cause an incorrect request. These are connected events, not automatically four independent failures. The first visible problem is also not proof of the internal root cause.

text

Condition: submission tool returned a rejection
Expected: report the rejection and retain the draft
Divergence: final response claimed successful submission
Propagation: user stopped monitoring the request
Consequence: approval work did not begin
Cause hypothesis: response generation ignored action status
Next evidence: inspect the exact input to response generation

Compare boundaries, including successful cases

Use concrete labels early: “claimed submission after rejection” is more useful than “bad reasoning.” Then compare incidents with similar executions that succeeded. Does the pattern depend on a particular tool status, a retry, a long conversation, or an ambiguous supplier?

Develop recurring categories only when their members share a meaningful behavior and a reason it is unacceptable. Split a category when it hides distinctions that would change judgment or intervention. Keep contradictory and disputed cases available; they can expose an overbroad failure definition.

Investigate causes with separate evidence

A trace can show that the response ignored a rejection. It may not show whether the cause was missing context, incorrect state mapping, conflicting instructions, or the model’s interpretation. Write these as competing hypotheses and identify an observation that could distinguish them.

Inspecting the assembled model input may reveal that the rejection was omitted. Replaying a fixed trace with corrected state can examine a proposed remedy. Avoid changing several components together and then attributing improvement to the most plausible explanation.

Leave an actionable failure record

Keep representative traces, successful contrasts, the violated expectation, inclusion and exclusion boundaries, observed consequences, and unresolved questions. Route findings to the right work: product clarification, an implementation investigation, missing instrumentation, evaluation coverage, or a criterion suitable for repeated checking.

A failure-discovery collection often overrepresents difficult cases. Use it to understand patterns and design follow-up work. Estimate live frequency through a separate sampling plan. Understanding why a category is useful should precede turning its count into a dashboard or a release threshold.

PUT THIS NOTE TO WORK

Write a trace-linked failure record

Choose one representative execution. Another reviewer should be able to recover the finding from the linked artefacts and understand what remains unresolved.

Retain the relevant input, state, retrieved material, proposed actions, confirmations, tool results, and final effects with versions.
State the applicable expectation and its owner. Separate confirmed failure, evidence gap, and unresolved product intent.
Identify the first demonstrated divergence and distinguish it from propagation and the user consequence.
Compare a successful contrast and record what does and does not belong in this failure category.
List competing explanations and the next evidence that could distinguish them before assigning a broad fix.

Open the copyable working record

Copy this blank record into your project and fill it with your own evidence. Keep links to the underlying work.

# Trace-linked failure record

Case ID and trace location:
Task, user, environment, and relevant versions:
Applicable expectation and owner:
Judgment: confirmed failure / evidence gap / intent gap

## What the evidence shows
Initial condition and relevant state:
Earliest demonstrated divergence and evidence link:
Propagation through later steps:
Observed user or system consequence:
Missing or contradictory observations:
Successful contrast and category boundary:

## What to investigate next
Cause hypothesis A and distinguishing evidence:
Cause hypothesis B and distinguishing evidence:
Proposed observation, experiment, or intervention:
Regression case and validation method:
Next decision, owner, and review date:

What you should leave with

Leave with a finding someone else can inspect and a specific next action: clarify intent, capture evidence, test an explanation, or validate an intervention.

A convincing trace establishes observed behavior within its scope. It does not automatically identify a root cause or estimate how often the failure occurs in production.

Adapted from our working notes on AI products and engineering. Explore the broader body of work at valery.tech ↗.

Back to technical notes