The disagreement may be about the product
Imagine a team building an assistant that suggests replies to customer questions. Engineering sees successful requests and acceptable response times. A support lead notices that some replies promise exceptions the company cannot offer. Product sees that agents keep rewriting the suggestions. Each person is observing something important. A green dashboard can coexist with a product that creates extra work or makes the wrong promise.
Asking one team to own quality does not settle these differences. The first task is to describe the behavior the product is meant to support. Which requests does it handle? What may it promise? When should it ask for help? What should a support agent be able to inspect? Until these expectations are explicit, people can use the same word, quality, while judging different products.
Why a score cannot own the decision
A score summarizes observations under a chosen rule. Deciding which observations matter and which tradeoffs are acceptable requires a product commitment. A team might improve average helpfulness while increasing a rare but consequential kind of policy error. Whether that is acceptable depends on the intended use, the people affected, and the controls around the answer. The average contains none of that authority. Asking an evaluation specialist to settle the release through a threshold can hide the underlying choice rather than resolve it.
This is the reason for distributing participation while keeping decisions explicit. Domain knowledge supplies distinctions the measurement may miss. Engineering establishes what the implementation can enforce and observe. Product connects the behavior to a supported task. The decision owner brings those contributions together and accepts the resulting commitment. Ownership becomes ineffective when it means merely receiving the report. An owner must also be able to narrow scope, request an investigation, delay a change, or accept a stated limitation. Responsibility without those options leaves the important decision somewhere else.
Assign ownership to decisions
A useful starting point is a small set of actual decisions. Who can change the supported scope? Who defines whether a reply applies a policy correctly? Who decides that a model change has enough evidence for release? Who responds when production exposes a failure? Different decisions can have different owners. Each needs a clear place where the evidence, authority, and consequences come together.
For the support assistant, a domain expert might define policy interpretation, an engineer might own system controls, and a product lead might own the proposed release scope. The release decision needs their relevant findings, along with a named person accountable for the commitment. Write down the decision, its owner, the necessary contributors, and the evidence expected. This is more useful than assigning a broad responsibility that everyone interprets differently.
Treat disagreement as information
Bring a few realistic interactions to the people who know the work. Ask them to judge the same cases independently before discussing the results. A disagreement may reveal an unclear criterion, missing context, a policy exception, or a product choice that has never been made. Preserve the reason for disagreement instead of immediately turning it into a majority vote.
Suppose one reviewer accepts a reply because it quotes the policy accurately, while another rejects it because that policy applies to a different customer tier. The useful result is a sharper condition: the system must establish which tier applies before offering the remedy. Update the expected behavior and examples accordingly. Some disagreements require an authoritative domain decision; others remain unresolved and should limit what the product currently promises.
Work through a disputed release
Return to the hypothetical support assistant. The proposed release adds answers about account refunds. Reviewers agree that the generated wording is usually clear, but disagree about a case involving a customer whose subscription changed halfway through the billing period. Treating the dispute as an annotation problem would invite a corrected label and another evaluation run. First establish whether the applicable refund rule is actually defined. If the rule is unresolved, changing the label would manufacture certainty that the organization does not have.
The domain owner could resolve the rule, or the release owner could exclude this situation from supported automated replies. Suppose the team chooses the second option. The commitment becomes specific: provide policy-backed drafts for unambiguous refund cases; identify a mid-period plan change and route it to a support specialist. Engineering then needs an observable way to recognize the condition, and evaluation needs both cases that should be routed and nearby cases that should remain answerable. The record should also explain the cost: more human handling and a narrower feature. A successful release is now judged against that explicit agreement, not against a larger promise the evidence never supported.
Give evaluation a route into the work
An evaluation function needs more than a dataset and a reporting deadline. Its findings must arrive where they can change discovery, implementation, and release decisions. A recurring policy error might require a clearer user interaction, different source selection, a software control, or a narrower supported task. The appropriate response depends on how the failure arose.
Keep the cases and observations close to those decisions. A useful finding explains what happened, under which conditions, why it matters, and what remains unknown. Preserve the relevant product and system versions so the team can revisit the evidence. Evaluation specialists can maintain sound measurement and investigation practices while product, domain, and engineering contributors remain involved in the judgments that require their knowledge.
Keep the learning loop owned as well
Production will introduce requests and conditions the team did not anticipate. Decide who examines these observations, how consequential failures are escalated, and who can change scope, implementation, or evaluation coverage. A collected trace creates little value if it never reaches someone able to act on it. A support complaint becomes useful engineering knowledge when its circumstances and consequences are understood.
Begin with one important decision and one shared review of real cases. A small team may need a short discussion and a clear written record. A larger organization may need explicit handoffs between functions. Scale the arrangement to the work. The aim is to make quality a capability that people can exercise together, with enough ownership that difficult questions lead to a decision and enough perspective that important failures remain visible.
Choose the lightest arrangement that can decide
There are legitimate alternatives to a separate evaluation function. A small product team may combine domain review, implementation, and evaluation in a shared working session. A larger organization may need specialists to maintain sampling methods and common tooling across products. Centralizing those capabilities can reduce duplicated work, but it can also distance the evaluator from the actual task. Embedding every evaluator can improve context while making methods inconsistent. The useful design question is where judgment depends on local knowledge and where shared expertise makes that judgment stronger.
The recommendation would change if the decisions were already clear and the recurring failures came entirely from a narrow technical defect. In that situation, rebuilding the operating model would add work around a problem that needs an engineering fix. Similarly, familiar reversible changes rarely justify a new committee. Examine whether the arrangement shortens unresolved decisions, preserves important dissent, and turns production findings into changes. If it mainly produces meetings and ownership charts, simplify it. The point of explicit ownership is an exercised decision right with visible reasoning, not a larger organizational structure.