AI adoption needs a budget for variability

The useful idea

Treat consequential variation as an operating expense and a release constraint. Fund the capacity to measure, review, and resolve it before increasing volume.

This week’s signal: measure what changes

On November 5, 2026, the Journal of the American Medical Informatics Association published a peer-reviewed framework for measuring repeatability and reproducibility in large language model outputs. Cathy Shyr and colleagues distinguish repeated runs under unchanged conditions from runs across deliberately changed conditions. Their semantic measures concern output meaning; their internal measures concern token-generation certainty. These describe variation, not a general guarantee of correctness or clinical suitability.

A different development makes that distinction operationally timely. In a November 3 update to its May 19 article, Google Research announced that approved users could integrate Computational Discovery through MCP into compatible agents. This is a vendor statement about access and integration, not independent validation of a downstream workflow. The November update should not be mistaken for a new publication date for the underlying research.

Together, these developments raise a practical question for decision makers: when AI becomes easier to connect to existing work, who pays for the variation that connection introduces? Metrosion’s assessment is that adoption plans need an explicit allowance for review demand and revalidation, alongside expected productivity gains.

Define the variation the business can tolerate

Different wording is often harmless. Different treatment of a mandatory exception may be consequential. Before choosing a similarity metric, identify which outputs must remain stable: an extracted amount, an escalation decision, a source reference, or a proposed action. A polished paraphrase and a changed decision should not consume the same risk allowance.

Consider a hypothetical assistant reviewing maintenance records for missing documentation. Two runs can produce different summaries while correctly flagging the same missing inspection. Conversely, nearly identical summaries can disagree about whether work should be held. Evaluate that decision explicitly. Semantic similarity can help organize review, but it cannot replace the task’s acceptance criteria.

Consistency also needs a correctness check. A system that repeats the same unsupported conclusion is predictable in an unhelpful way. Keep agreement, correctness, and consequence visible as separate dimensions when presenting results to an investment or release committee.

Translate uncertainty into review capacity

A synthetic planning example makes the economics concrete. Suppose a team handles 1,000 records a day and each escalated record requires six minutes of review. At a 5 percent escalation rate, that is five reviewer-hours daily. At 15 percent, it is fifteen hours. These are illustrative assumptions, not measured Metrosion results. Neither estimate includes supervision, breaks, or investigating unusually difficult cases.

The staffing question is therefore about the range and concentration of demand. A manageable daily average can conceal a queue that grows during a shift change or a burst of difficult records. Measure review arrivals and resolution times by task class and operating period. Forecast what happens when volume increases before interpreting time saved on routine work as capacity released.

There is a tradeoff: a cautious system may escalate more often while preventing costly unsupported decisions. Reducing escalation is useful only if the cases that proceed remain acceptable. Agree on service targets, reviewer availability, and unacceptable outcomes together; a lower inference bill cannot settle that decision.

Make changes comparable

Repeat a selected set of cases under a recorded configuration, then change one planned condition at a time where feasible. A model revision, retrieval update, tool integration, or prompt adjustment can each alter behavior. Preserve the inputs and configuration references needed to explain the comparison, with access controls and retention limits appropriate to the information involved.

The authors’ accompanying llmReproScore repository makes a useful practical limitation explicit: semantic metrics can use text outputs and embeddings, while internal metrics require token-level log probabilities that some endpoints do not expose. Choose measurements supported by the system you actually operate. Do not replace missing evidence with a model’s assertion that it is confident.

Repeated outputs from one case also do not create a large collection of independent business cases. The repository distinguishes descriptive repeated-run scores from inference using independent analysis units. In an adoption review, show both case coverage and repetition. Hundreds of runs on a narrow example cannot establish performance across a diverse workload.

Give adoption a measurable gate

Our recommendation is a short evidence plan owned jointly by engineering and operations. Select one workflow and define its consequential decisions, acceptable variation, reviewer capacity, and limits on automatic action. Use synthetic or appropriately authorized test material; no real patient information is needed for the planning exercise described here.

Compare the current and proposed configurations on the same cases, inspect consequential disagreements, and estimate review demand across plausible volumes. Record exclusions and unresolved uncertainty. Set the conditions for proceeding, limiting volume, or returning to the previous configuration before seeing the final results. Those conditions should reflect business consequences rather than a universal similarity threshold.

Protect measurement capacity in the budget. More integration points and faster release cycles can expand the work required to keep evidence current. A staged rollout preserves the option to learn without committing the organization to an unsupported staffing assumption. The useful next step is a meeting with the workflow owner and review lead: choose one planned change, identify the evidence it would invalidate, and price the work needed to restore confidence.