TECHNICAL NOTE / 5 MIN READ

Build retrieval you can inspect and explain

Preserve the path from source material to working context so a team can understand omissions, test transformations, and verify the resulting task.

VALERY · PRINCIPAL, EML SYSTEMS

Retrieval
Context engineering
Provenance

THE PRACTICAL ANSWER

What must an inspectable retrieval path retain?

Start with the consuming task and preserve the route from source occurrence to assembled input: source identity and revision, representation, query, candidates, selection, and transformations. Evaluate whether the required information survives and whether the resulting task succeeds.

When to use this

Use this when designing a retrieval route or investigating a missing qualification, stale source, or unsupported answer. It helps locate the next comparison; relevance alone cannot establish task success or source authority.

Use the checklist and record

Define what will consume the context

Retrieval begins with an information need and a next operation. A policy reviewer may need original passages with dates and exceptions. An extraction step may need a document and a strict field schema. A user exploring a disagreement may need several conflicting sources without an automated answer.

Write down the task, the material required, the available sources, and what would count as sufficient evidence. This creates a basis for evaluating context selection. Otherwise a plausible answer can conceal the fact that the system never retrieved the material the user wanted.

Give source material a recoverable identity

A retrieved chunk should resolve to a particular source occurrence. Preserve a source identifier, revision or snapshot, structural location, and the original text. When indexing a derived view, such as questions generated from a paragraph, retain the mapping back to that paragraph.

Parsing and segmentation are consequential steps. A table can lose its headers; a qualification can become detached from the rule it limits. Inspect the searchable representation before blaming ranking. No downstream reranker can recover information that corpus preparation discarded.

json

{
  "sourceId": "purchasing-policy",
  "sourceRevision": "rev-12",
  "locator": { "section": "4.2", "paragraph": 3 },
  "view": "original-paragraph",
  "retrievalRoute": "literal",
  "query": "supplier approval exception",
  "transformations": [],
  "selectedFor": "review-approval-requirements"
}

Keep acquisition, selection, and assembly visible

Acquisition produces candidate material. Selection decides what to retain. Assembly orders, formats, expands, or compresses it for a consumer. These decisions may share one implementation, but keeping their outputs inspectable lets the team localize a problem and replace one method without rebuilding the whole pipeline.

Retain the actual query, route, candidate set, selection reasons, and input manifest. A relevance judgment should identify the task it concerns. Similarity, source authority, freshness, and support for a particular claim are different properties; one score should not silently stand in for all of them.

Test one meaningful change at a time

Choose real scenarios that expose different difficulties: exact vocabulary, paraphrases, conflicting versions, repeated text, and missing subject matter. Start with a small route whose assumptions are visible. Literal matching can serve one task; semantic acquisition may be appropriate immediately for another.

Compare a proposed addition against that route. If testing model-based filtering, inspect both retained and excluded candidates. If testing compression, examine whether exceptions and provenance survive. Keep the source snapshot, task, and comparison budget sufficiently stable to interpret the difference.

Missing source: examine collection scope and synchronization.
Missing candidate: examine representation and acquisition.
Missing qualification: examine expansion, selection, and compression.
Unsupported answer: inspect the assembled input and its use.

Make interventions part of explicit state

A reviewer may expand a passage, change source scope, reject a relevance suggestion, or correct the interpretation of a request. Preserve the original result, the revised artifact, and the reason. A downstream answer should identify which version it consumed.

These interventions help the current task and can suggest a policy for later tasks. They are different claims. Including a preceding paragraph because it defines an acronym does not establish that every retrieved paragraph should always include its predecessor. Test the proposed general rule across other cases.

Verify the actual input and the final job

Review the context that reaches the consumer, not only the material available earlier in the pipeline. Check required evidence, unresolved conflict, source currency, formatting, and task limits. Changing sources or requirements can make an earlier sufficiency judgment stale.

Then evaluate the complete task separately. Include missed evidence, incorrect exclusions, reviewer effort, latency, and cost alongside retrieval measures. Provenance makes these investigations possible; it does not automatically establish that a source is correct or that the final answer used it well.

PUT THIS NOTE TO WORK

Record one inspectable retrieval comparison

Trace a meaningful task through the existing route before adding another ranking, filtering, or compression step.

Define the next consumer, the information it needs, and the evidence that would be sufficient for the task.
Verify that source identity, version, location, permissions, and material qualifications survive representation.
Retain acquisition, selection, and assembly outputs so an omission can be located.
Compare one consequential change on stable source snapshots and realistic cases; inspect removed material as well as retained material.
Review the assembled input and accepted task outcome alongside human effort, latency, cost, and unresolved gaps.

Open the copyable working record

Copy this blank record into your project and fill it with your own evidence. Keep links to the underlying work.

# Retrieval comparison record

Consumer and task:
Required information and sufficiency criteria:
Source scope, permissions, and snapshot:
Source IDs, revisions, and original locations:

## Existing route
Representation and transformations:
Query and acquisition method:
Candidate manifest and selection reasons:
Assembly, expansion, or compression:
Actual input manifest and source links:
Observed omission or unsupported use:

## Proposed comparison
One change and the hypothesis it tests:
Cases, important boundaries, and comparison conditions:
Required information retained or lost:
Accepted task result and human review effort:
Latency, operating cost, and maintenance implications:
Recommendation, remaining gaps, owner, and review date:

What you should leave with

Leave with a recoverable route, an explained information gap or justified change, and evidence about the task the context must support.

Provenance makes the path inspectable. It does not make a source correct, grant access, or guarantee that the model will use the information appropriately.

Adapted from our working notes on AI products and engineering. Explore the broader body of work at valery.tech ↗.

Back to technical notes