Production readiness and evaluation as decision evidence

1. Readiness needs an object

A prototype has produced convincing answers. Its architecture is documented, a test set looks promising, and a team wants to ship. “Is it production ready?” sounds like a technical question with one answer. It is incomplete until the team states the commitment being considered. A draft assistant used by a small internal group and a service that commits business actions on customers’ behalf create different obligations.

Our position is that readiness belongs to a specified product promise and operating arrangement. The same implementation can be adequate for one use and inadequate for another. The promise includes the supported work, expected behavior, affected actors, operating limits, and response to uncertainty. It also includes someone accepting responsibility for maintaining those conditions.

This makes readiness revisable. A new source collection, broader user population, changed action authority, or different latency obligation can require another judgment even when the model is unchanged. Conversely, a model replacement need not change the promise, although it requires evidence that the replacement continues to support it.

2. Separate the promise, implementation, evidence, and decision

Four objects deserve separate treatment. Product intent states what should happen. The implementation describes how the current system attempts it. Evidence records observed behavior under particular conditions. A decision states what the team will do, which uncertainty it accepts, and who owns the consequences. Combining these objects makes it easy for current output to become its own definition of correctness.

Consider an approved reference answer. It may demonstrate a useful style while leaving the underlying policy ambiguous. An evaluator that rewards similarity can favor that example without establishing whether it is appropriate in a changed situation. The team must decide which facts, rules, and judgments make the reference binding, illustrative, or open to revision.

This broader engineering scope has precedent. Breck and colleagues’ ML Test Score proposes tests and monitoring needs for production ML beyond offline prediction performance. We draw on that distinction, without assuming its supervised-learning rubric transfers unchanged to language-model workflows or validates our proposed commitment record.

Supporting reading: [1] Breck et al. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction.

3. Why an evaluation result needs an evidence chain

A score becomes decision evidence through a chain of justified relationships. The cases must represent relevant situations. Executions must reflect the system under consideration. Captured artifacts must establish what happened. Criteria must express the applicable expectation. Evaluators must apply those criteria adequately. Finally, the interpretation must stay within the conditions examined.

A weak link changes what the result can support. A diagnostic set may reveal a possible failure without measuring its frequency. A model-based judge may consistently reward a plausible response that violates a domain rule. A complete answer may still require so much verification that the proposed workflow provides little benefit.

Evaluation should therefore preserve both findings and limitations. We propose attaching each material result to the decision it informs, relevant behavior, case or sample definition, implementation versions, criteria, judgment method, and unresolved uncertainty. This need not require a new platform. A small record can make the reasoning inspectable before automation becomes worthwhile.

4. Write a promise that engineering can support

Begin with the people and work in scope. Describe successful completion, supported situations, material ambiguity, unsupported requests, and failures that must be prevented or contained. State which actions require confirmation, which data sources govern, and what observation and recovery must exist. Distinguish desired business effects from behavior the team directly undertakes to provide.

NIST’s AI Risk Management Framework 1.0 connects governance, context mapping, measurement, and management across the lifecycle. It supports treating measurement within a wider responsibility structure. It does not prescribe this paper’s particular production-promise format or establish a universal release threshold.

A useful initial promise can be small. Reduce the situations supported before silently dropping the controls required within them. If the team cannot explain how uncertainty is exposed or a failed action is recovered, a polished interface does not resolve the gap. It only makes the unresolved commitment easier to distribute.

Supporting reading: [2] NIST (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0).

5. A hypothetical purchasing assistant

Imagine an internal assistant that turns an employee’s message into a purchase-request draft. The proposed initial scope covers ordinary office equipment from an approved supplier list. Employees inspect the draft and confirm submission. The assistant may recommend a matching supplier, but it must expose material ambiguity and report a submission failure accurately.

The first investigation finds acceptable drafts for unique supplier references and unreliable behavior when two active suppliers share a name. These are hypothetical observations used to reason about a decision, not results from an engagement. They could support several next steps: improve matching, use a structured selector, narrow the initial scope, or reconsider the interaction.

Suppose the team chooses the selector and a narrow release. The commitment must now include how ambiguous requests reach it, whether users notice unresolved fields, whether validation prevents unsupported identifiers, and what happens after the submission service rejects a request. An interpreter score alone cannot establish these end-to-end properties.

6. Design evidence for the proposed exposure

Coverage should follow that commitment. Include unique supplier matches, several matches, inactive records, missing fields, corrections, retries, and rejected submissions where they affect promised behavior. Pair the user input with the minimum state needed to create each situation. Identical wording can exercise different behavior under different supplier records.

Use a comparison to examine the proposed change while retaining cases that protect existing behavior. Keep source snapshots, model configuration, tools, and relevant environment conditions sufficiently stable to interpret differences. Repeat executions when variability could affect the decision. Preserve unsuccessful and unjudgeable runs rather than making them disappear from the denominator.

Observe user review separately. Can employees recognize a wrong supplier, correct the draft, and complete the request? The interaction might satisfy the behavioral rule while increasing effort. These findings concern different claims and may lead to different interventions. A technically successful selector is a candidate solution until the whole workflow is examined.

7. Use controls and evaluation for different responsibilities

Exact action boundaries should be implemented where they can be enforced. A submission service can require a valid confirmation token, validate supplier identity, and use an idempotency key. Those controls do work during execution. An evaluation can examine whether the system preserves them under relevant conditions, but observing past compliance does not itself prevent the next invalid action.

Semantic behavior may require another judgment method. A structured event can establish that a tool rejected submission. Deciding whether a natural-language reply falsely implies success can require a reviewed criterion and a validated semantic evaluator. Keep those judgments separate so disagreement can be investigated.

The placement also matters. A judge used asynchronously for monitoring has different latency and failure requirements from one blocking a live request. A release decision should account for evaluator uncertainty, missing evidence, the consequence of a false pass, and the operating response when a check cannot decide.

8. A practical decision record

We propose one short record per consequential commitment, with links to supporting evidence. The record should identify what would cause the team to reconsider. The purpose is to preserve a reviewable argument, not to complete a standard document regardless of relevance.

After release, compare observed behavior and conditions with the promise. Route findings according to what needs to change: implementation, supported scope, product intent, criteria, or the capability to capture and judge evidence. Maintain the record as those decisions evolve.

• Commitment: supported users, situations, behavior, and operating limits.
• Evidence: cases or samples, exact versions, criteria, findings, and limits.
• Controls: enforced boundaries, failure detection, fallback, and recovery.
• Decision: accepted uncertainty, accountable owners, and exposure policy.
• Reconsideration: signals that trigger investigation, restriction, or withdrawal.

9. Counterarguments: does this slow down learning?

One objection is that defining a promise before release adds process when rapid experimentation is needed. The objection is valid if every experiment receives a production-sized review. Our proposal is proportional: identify the actual commitment and choose the smallest credible evidence for it. A contained experiment can accept more uncertainty while retaining clear observation and failure handling.

Another objection is that the expected behavior cannot be fully specified. We agree. The promise should make material expectations and uncertainty visible without pretending to enumerate every possible input. Cases and production observations can reveal that the intended behavior itself needs revision. That is a product finding rather than an embarrassment to remove from evaluation.

A third objection is that a strong aggregate score is sufficient for a commercial decision. It may be sufficient evidence in a narrow, well-understood setting. The difficulty is explaining what the score covers, how it was produced, and whether material failure slices are hidden. A decision record can be short when those answers are already clear.

10. Limitations and a test of the proposal

This framework does not prove that a product is safe, valuable, or reliable in every situation. It does not replace security engineering, usability research, legal obligations, economic analysis, or ordinary software testing. It cannot create information that the team lacks, and documentation can become a ritual disconnected from actual authority.

Its benefit therefore needs local evaluation. A team could examine whether the record exposes consequential assumptions earlier, changes a release or scope decision, or improves recovery after a failure. The cost includes preparation, review, maintenance, and delays caused by unclear ownership. Finding no useful effect would justify simplifying or abandoning parts of the method.

Our concluding claim is deliberately bounded: production decisions become easier to examine when the promise, evidence, controls, and accepted uncertainty are explicit. Evaluation earns its place by improving those decisions and the product learning that follows. It remains a maintained engineering capability whose adequacy must itself be questioned.