You may have a promising capability, a product already in use, or a team under pressure to deliver more with AI. The immediate question is usually concrete: what should we invest in, can we release this, why does it fail, or how do we make progress without losing control of the system?
These questions connect product behavior, architecture, evidence, and the people responsible for the work. We start with the decision you need to make and investigate the parts of that system that matter. The situations below are starting points for a conversation. Similar symptoms can have different causes; understanding your conditions comes before prescribing an intervention.
THE CLIENT’S CASE FOR ACTING
Why this work may be worth doing
A recognizable symptom becomes a business priority when it has a material consequence, evidence in your own work, and a decision that investigation can improve. Use these three tests to decide where attention belongs and whether specialist support would add value.
01
A consequence that matters
Identify what the unresolved question puts at stake: development effort, a customer commitment, recovery work, release delay, or the ability to maintain a system. Follow the causal path from the observed problem to that consequence. Its importance depends on your exposure, alternatives, and the cost of acting.
02
Evidence in your own work
Look for representative cases, execution records, review effort, repeated decisions, or customer observations that establish the concern locally. Distinguish an isolated event from a recurring pattern. Where evidence is missing, first decide whether collecting it is justified by the consequence and the uncertainty it could resolve.
03
A decision the work can improve
Name the choice that better understanding could change: invest, narrow scope, release, redesign, alter ownership, or stop. Define a useful result and an alternative response before beginning. An investigation earns its effort when its findings can improve that choice enough to justify preparation, implementation, and ongoing maintenance.
Start with your evidence
Use these situations to test applicability to your organization. Bring a decision, a few representative cases, and what you currently know. Together we can establish whether the consequence is material, which explanation needs testing, and what improvement would justify further work.
01We need to invest in AI, but the right product is unclear.Founder, Head of Product, or CTO choosing the next investment
What is at stake
If a proposal improves an isolated AI task but leaves the complete job unchanged, development can consume capacity without relieving the customer’s constraint. Early investigation is valuable when it can redirect that commitment.
Evidence to establish the problem
Examine current workflow observations, workarounds, customer requests, and where effort or delay occurs. Check which assumptions connect each candidate solution to the intended improvement and whether meaningful alternatives have been considered.
What improvement would mean
Compare the complete job against its baseline: accepted outcomes, user effort, elapsed time, and relevant operating cost. Before implementation, record which investment or scope decision changed because of the evidence and why.
02The demo is convincing. We cannot yet justify a release.CTO, engineering lead, or product owner responsible for launch
What is at stake
When people rely on behavior that has no clear support boundary, exceptions can become unplanned support work or broken commitments. The business question is which exposure the team can support with its current evidence and controls.
Evidence to establish the problem
Inspect the proposed users and tasks, observed behavior outside demonstration cases, unresolved release questions, and the actual detection and recovery paths. Establish which gaps affect the intended release rather than treating every possible concern as equally urgent.
What improvement would mean
Evaluate readiness against a bounded promise. Track consequential failures, recovery effort, support load, and whether agreed release conditions hold. A narrower supported release can be a useful result when a broader one lacks evidence.
03We have evaluations, but important decisions still rely on intuition.AI engineering lead, product lead, or evaluation owner comparing changes
What is at stake
If a score rewards behavior unrelated to the user’s job, improving it can direct effort toward the wrong work. If its meaning remains disputed, maintaining the evaluation may leave the original release or architecture choice unresolved.
Evidence to establish the problem
Review a recent decision alongside the scores, cases, criteria, and expert judgments used to make it. Look for consequential disagreements, hidden regressions, or conclusions stronger than the sample supports. Establish whether the mismatch actually changed a choice.
What improvement would mean
Assess whether revised evidence distinguishes the alternatives and changes prioritization. Follow relevant product behavior and complete-job outcomes alongside evaluator error, review effort, and maintenance cost. A higher aggregate score alone is insufficient evidence of benefit.
04Important failures keep appearing, and fixes do not explain them.Engineering manager, technical lead, or service owner facing recurring incidents
What is at stake
When a fix targets the visible answer while the cause lies elsewhere, the same condition can survive and create further repair work. A more expensive redesign can also be misdirected if its explanation has not been tested.
Evidence to establish the problem
Collect representative incidents, related successful executions, the applicable expectation, and the interventions already attempted. Determine whether the records support reconstruction, whether a pattern recurs, and which missing observations prevent competing explanations from being distinguished.
What improvement would mean
Track recurrence under comparable conditions, time spent diagnosing and recovering, and regressions introduced by intervention. Judge the improvement against confirmed failure behavior and user consequences; better instrumentation may initially reveal problems that were previously invisible.
05Every team has a different definition of quality, and decisions stall.VP Engineering, Head of Product, or leader coordinating AI delivery
What is at stake
When a recurring decision has no clear owner or lacks necessary domain input, work can wait for escalation or restart after late disagreement. The consequence depends on which commitments are blocked and whose judgment is missing.
Evidence to establish the problem
Trace recent delayed or reopened decisions. Identify participants, evidence requested, changing criteria, handoffs, and where authority became unclear. Separate legitimate new learning from avoidable repetition before attributing delay to the organization’s structure.
What improvement would mean
Observe time to a supported decision, repeated reviews, escalations, and whether material concerns are resolved before commitment. Preserve decision quality and domain participation as guardrails. A shorter meeting or fewer reviewers does not by itself establish improvement.
06Coding agents produce changes faster than we can confidently accept them.Engineering director, platform lead, or maintainer expanding coding-agent use
What is at stake
If implementation output grows faster than review and understanding, more work can remain unfinished or require correction. The investment question concerns accepted, maintainable changes and the human effort needed to produce them.
Evidence to establish the problem
Follow comparable changes from task framing through acceptance and relevant rework. Record preparation, intervention, review, waiting, and validation. Check whether observed difficulties arise from the agent, repository context, task selection, or an existing workflow constraint.
What improvement would mean
Compare accepted completion, total human effort, review delay, unintended changes, and subsequent defects for the relevant task population. Include execution and tooling costs. Require accountable maintainers to retain enough understanding to operate and evolve the affected system.
We need to invest in AI, but the right product is unclear.
Leadership wants an AI initiative. Several ideas have convincing demos, yet the team cannot explain which customer or operational condition each would improve, or what would justify a larger investment.
You want a direction that earns development effort. The decision may concern a new product, an internal workflow, or an existing feature. You need a credible connection between the proposed capability and useful change.
What may be underneath
One possibility is that the solution was chosen before the problem was understood. Another is that the task looks easy to automate while the surrounding job still depends on exceptions, tacit knowledge, or downstream coordination. A successful generated answer establishes only part of the case for a product.
We examine the current work, meaningful alternatives, and assumptions connecting the intervention to the intended result. For example, faster report drafting may contribute little if verifying the report remains the dominant effort. Leaving those assumptions implicit can commit the team to an architecture and roadmap before it knows which improvement matters.
Questions worth answering first
Who experiences the problem, and what happens in their current workflow?
What should change, and what observation would show that it changed?
Which assumption could invalidate this proposal before substantial implementation?
What we work on together
Map the complete job, its current constraints, and the baseline against which an improvement would be judged.
Compare meaningful solution options, including simpler software or changes to the work where relevant.
Produce an assumption map, a small experiment plan, and a written continue, change, or stop recommendation.
The decision this should unlock
First useful decision: which uncertainty deserves the next investment, and what result would change direction. A focused product assessment fits this stage. Hands-on work can follow when a bounded experiment or prototype is the credible next source of evidence.
SITUATION / 02
The demo is convincing. We cannot yet justify a release.
A pilot is attracting interest, a launch date is approaching, or colleagues have begun relying on a prototype. Selected examples look good, but supported situations, unacceptable failures, and operational responsibilities remain unclear.
You want to make a release commitment the team can actually support. That means knowing where the system is useful, where its limits belong, and what happens when real use departs from the demonstration.
What may be underneath
The gap may be missing evidence, an overly broad product promise, or a system whose controls do not match the consequences of its actions. A prompt describing the desired behavior does not establish that permissions, confirmations, or state changes are enforced. Each important claim needs an appropriate mechanism and evidence.
We make the proposed production scope explicit and inspect the path through success, uncertainty, and failure. A narrow release may be supportable even when the larger ambition is not. Letting the prototype become a service by gradual reliance can otherwise create obligations before ownership, observation, recovery, and support are ready.
Questions worth answering first
Which users, tasks, inputs, and operating conditions are we committing to support?
What evidence challenges the promise beyond the examples used to build it?
Who detects a consequential failure, limits exposure, and restores a usable workflow?
What we work on together
Write a bounded production commitment covering supported behavior, limits, failure responses, and responsibilities.
Review critical architecture and action boundaries; identify controls, observations, and recovery paths that need implementation.
Build a coverage and evidence plan, prioritized readiness findings, and a release recommendation with explicit remaining uncertainty.
The decision this should unlock
First useful decision: release a defined scope, constrain the promise, or investigate a named gap. A readiness assessment can establish that decision. A hands-on engagement fits when the team needs evaluation, instrumentation, or system changes to make the commitment supportable.
SITUATION / 03
We have evaluations, but important decisions still rely on intuition.
A dashboard improves after a model or prompt change, yet reviewers disagree about whether the product is better. Benchmarks, automated judges, and test sets exist, but release and investment discussions keep returning to isolated examples.
You need evidence that helps choose between alternatives, prioritize work, and explain a release decision. Maintaining more evaluations is difficult to justify when their results do not resolve the questions the team actually faces.
What may be underneath
Possible causes include a convenient metric standing in for the product objective, cases that miss supported situations, or a judge whose errors are poorly understood. An overall improvement can conceal a consequential regression in one workflow. A correct answer can also leave the complete user job slower or harder.
We trace an important decision backward through its claims, criteria, cases, observations, and judgment mechanism. The issue may require better coverage, clearer product intent, stronger measurement, or a different kind of experiment. Leaving the connection unresolved can reward optimization of a score while the team remains unable to defend what changed for users.
Questions worth answering first
Which decision should this evaluation inform, and what result could change it?
Do the criteria reflect intended behavior independently of the current implementation?
Where do reviewer disagreement, evaluator error, exclusions, and unknown outcomes appear?
What we work on together
Map decisions to claims and evidence; identify unsupported conclusions and measures that need qualification.
Review representative traces, important segments, case selection, and judge behavior against relevant domain judgment.
Deliver a revised evaluation design, an interpretable comparison, and guidance connecting results to release or development decisions.
The decision this should unlock
First useful decision: which existing evidence can be trusted for the next choice, and which gap needs attention first. A focused evaluation assessment fits. An implementation engagement can establish a maintained evaluation system; advisory can keep its findings connected to recurring decisions.
SITUATION / 04
Important failures keep appearing, and fixes do not explain them.
Users report failures the team cannot reproduce, a prompt patch helps one example and breaks another, or an incident discussion ends with a broad label such as hallucination rather than an actionable account of what happened.
You want to identify what needs to change and avoid repeatedly treating the same symptom. You also need to distinguish demonstrated failures from plausible explanations before committing to a model migration, architecture rewrite, or growing collection of exceptions.
What may be underneath
The visible answer may be the end of a longer failure path involving source data, retrieval, state, tool results, or interface behavior. Missing execution evidence can prevent a verdict. An unclear product expectation can make reasonable reviewers disagree about whether the observed behavior was unacceptable at all.
We reconstruct representative executions and locate the earliest divergence the evidence supports. Competing cause hypotheses remain separate from observed behavior. Successful contrast cases help identify the boundary of a pattern. Without that work, local patches can accumulate while the underlying condition survives, and the team may invest in the wrong layer of the system.
Questions worth answering first
Can another reviewer reconstruct the relevant inputs, state, actions, and consequences?
Which expectation was violated, and where does the trace first demonstrate the departure?
What evidence would distinguish the competing explanations for this pattern?
What we work on together
Assemble trace-linked failure records, successful contrasts, and explicit evidence or product-intent gaps.
Develop application-specific failure categories with inclusion boundaries, consequences, and unresolved cause hypotheses.
Produce a prioritized investigation and intervention plan, with targeted regression cases and instrumentation changes where justified.
The decision this should unlock
First useful decision: whether the next action is a product clarification, better observation, a discriminating experiment, or a demonstrated fix. A failure-analysis assessment fits an unclear pattern. Hands-on engineering fits a well-supported intervention and the feedback loop needed to assess it.
SITUATION / 05
Every team has a different definition of quality, and decisions stall.
Product, engineering, and domain experts repeatedly reopen the same questions. An evaluation team reports scores, but nobody has clear authority over supported behavior, acceptable failure, or the evidence required for a release.
You want decisions to move without depending on repeated escalation to a few specialists. The organization needs appropriate expertise in the conversation, clear responsibility for the commitment, and learning that survives beyond the people present in one meeting.
What may be underneath
This may be a language problem, missing domain participation, fragmented decision ownership, or a handoff that turns evaluation into a reporting service. The same word can conceal several claims: technical reliability, policy correctness, usability, and workflow benefit. Assigning all of them to one function can leave important knowledge outside the decision.
We begin with recurring decisions and examine who contributes, who decides, and which artifacts carry the evidence. Responsibilities follow the actual work. Continuing with vague ownership can create duplicated reviews, delayed releases, unexamined assumptions, and recurring debates whose conclusions never become part of how the product is developed.
Questions worth answering first
Which decisions recur, and who is accountable for each resulting commitment?
Where does domain knowledge enter product design, evaluation, and failure review?
How do production observations change scope, architecture, criteria, or working practices?
What we work on together
Map decision ownership, necessary contributors, evidence requirements, and gaps across the product lifecycle.
Use shared case review to clarify disputed behavior, operational definitions, and unresolved product choices.
Establish lightweight decision records, review routines, and a production-learning path with named responsibility.
The decision this should unlock
First useful decision: which recurring decision needs clearer ownership and a shared evidence path. An operating-practice assessment fits cross-team friction. Principal advisory can support the new routines as real decisions expose where the arrangement needs to change.
SITUATION / 06
Coding agents produce changes faster than we can confidently accept them.
Implementation drafts arrive quickly, while review queues, corrections, or unfamiliar complexity grow. The team is being asked to expand agent use but cannot yet explain its effect on accepted work, human effort, or maintainability.
You want faster delivery of useful changes with people who can still understand and evolve the system. You need to know which improvements to the agent, repository, or workflow will actually relieve the current constraint.
What may be underneath
The bottleneck may have moved from writing code to framing tasks, supplying context, validation, or review. An agent can also expose existing weaknesses in documentation, ownership, or tests. Faster implementation does not establish which of these explanations applies, and a plausible explanation of a patch is only a starting point for investigation.
We follow representative changes through preparation, execution, intervention, acceptance, and relevant rework. Core product changes need a maintained understanding of the constraints and design. If changes keep building on unresolved comprehension gaps, the team can lose the basis for judging the next proposal while throughput appears healthy.
Questions worth answering first
What counts as an accepted change, and how much human work precedes it?
Which context, tools, or checks are repeatedly missing when the agent struggles?
Can accountable maintainers explain the affected behavior, constraints, and failure paths?
What we work on together
Map the change workflow and distinguish execution speed, waiting time, human effort, and acceptance quality.
Review repository orientation, task boundaries, validation, and ownership against concrete sessions and resulting changes.
Run a scoped comparison of a proposed improvement and deliver maintainable guidance, tooling changes, and acceptance criteria.
The decision this should unlock
First useful decision: which constraint to address before increasing agent throughput or authority. A focused workflow assessment fits. Hands-on work can improve the repository environment and validation; ongoing advisory can help maintain acceptance standards as tools and tasks change.
THE MOTIVATION BEHIND OUR PRACTICE
Why EML works across the whole system
Our motivation is to help organizations become more capable of engineering useful AI products while the practice itself continues to develop. An isolated fix can address an immediate failure. Understanding why the failure was possible can also improve how a team frames problems, designs systems, evaluates behavior, and learns from real use. Both levels matter.
Evaluation is central because claims about behavior need evidence. That evidence becomes useful when it can change a product decision, an architectural choice, a release commitment, or an operating practice. We bring these concerns together around the client’s work. The aim is a better decision now and a reusable capability afterward, with reasoning your team can examine, challenge, and maintain.
FROM RECOGNITION TO A USEFUL NEXT STEP
Start with the decision that matters next.
A useful engagement begins with your situation, the commitment under consideration, and the evidence already available. You do not need to arrive with a diagnosis. A specific concern, a representative workflow, or a decision that keeps getting postponed is enough to establish a starting point.
We agree on the question, the scope, and reviewable outputs before expanding the work. Findings can justify implementation, a smaller experiment, a narrower promise, or stopping an approach. The engagement should leave your team able to understand the reasoning and use what it learns.
Focused assessment
Investigate a bounded product, architecture, evaluation, or operating question. Receive evidence-linked findings, explicit uncertainties, and a prioritized recommendation for the next decision.
Hands-on engagement
Work through a justified intervention alongside your team: a prototype, system change, evaluation capability, or production-learning path, with implementation and knowledge transfer shaped to the scope.
Principal advisory
Bring an independent perspective to recurring consequential decisions. Review evidence, assumptions, ownership, and tradeoffs as the product develops, while keeping responsibility for the system clear.