Legacy modernization needs behavioral evidence

The useful idea

Use stronger coding models to accelerate a bounded migration, with independently defined behavior checks and a controlled path back to the existing system.

A promising result with a defined scope

On October 19, 2026, OpenAI published new evaluations covering software construction across 20 programming languages and migration of COBOL programs. It reported 97.9% accuracy for GPT-6.1 Sol at higher reasoning effort on its COBOL migration evaluation, where success requires all hidden output-parity tests to pass. These are vendor-reported benchmark results, not a measured success rate for enterprise migrations.

The accompanying MultiLangBench repository makes part of the evaluation design inspectable. Its database task uses an initial database, a candidate query, and a validation query executed by a reference SQLite implementation. The README also describes a smaller default test subset. A team reproducing results should record exactly which tests it ran.

Our interpretation is that better language coverage warrants fresh, carefully bounded experiments. The useful question is whether a model can help preserve the specific behaviors an organization depends on, at an acceptable total review and migration cost.

Inventory the behavior people rely on

Before choosing a target language, we recommend documenting observable behavior at the boundary of the proposed migration. Include outputs, state changes, error responses, ordering, and interactions with surrounding systems. Add examples from the people who resolve operational exceptions; those cases often explain why an apparently awkward branch exists.

Consider a hypothetical nightly reconciliation process. A rewritten program produces the same total as the existing job, but changes the order of records in an exception file. A downstream operator may rely on that order to match records manually. Comparing only the total would miss a real workflow regression.

The reference system is useful evidence of current behavior, but it is not automatically the authority on intended behavior. When a discrepancy reveals a suspected old bug, record it separately. Have the business owner decide whether to preserve or deliberately change it. Combining modernization with undocumented corrections makes acceptance harder to reason about.

Build an independent comparison path

We recommend giving the coding agent a development test suite while reserving additional acceptance cases for an independent reviewer. Include ordinary traffic shapes, boundary values, incomplete records, repeated requests, and failure recovery. Define expected outcomes before inspecting the generated implementation whenever practical.

For the reconciliation example, run both versions against isolated copies of the same approved, appropriately protected dataset. Capture the full output and state transition. Normalize only differences explicitly judged irrelevant, such as a diagnostic timestamp. Every normalization rule should be reviewable, because an overly broad comparison can hide a meaningful mismatch.

SQLite’s own testing documentation provides useful background here: it describes fault-oriented checks including memory exhaustion, I/O failures, and crashes, alongside other testing methods. It is an established engineering reference, not a new announcement this week. Our recommendation is to borrow the discipline of testing adverse conditions when planning migration acceptance.

Treat cutover as its own engineering problem

Passing a comparison suite is one milestone. A production transition also needs a plan for data ownership, concurrent changes, and failure recovery. Start with a boundary that permits observation of the new implementation before it becomes authoritative. Shadow execution must isolate writes and external side effects so a comparison cannot create duplicate business actions.

In the hypothetical reconciliation migration, the existing job could remain authoritative while the candidate writes results into a separate comparison store. Review discrepancies by business impact instead of averaging them into one score. A single unexpected state change may outweigh thousands of matching rows.

Define the rollback boundary before cutover. If the new version changes stored data in ways the previous version cannot read, switching executable versions is insufficient. A reviewed recovery procedure should explain how to restore or reconcile state, who can authorize that step, and how operations will know which results are authoritative.

Measure the whole migration effort

For a first pilot, select one module with a clear interface, a knowledgeable owner, and representative test data. Set a fixed review budget and a list of unacceptable divergences. Ask the agent to produce both the proposed implementation and an explanation of behaviors that it could not confidently interpret.

Track time spent clarifying requirements, reviewing generated changes, investigating mismatches, and rehearsing recovery. Model usage cost is only one part of the economics. A fast rewrite that leaves difficult exceptions unexplained may shift work into a more expensive operational phase.

The concrete next step is a small migration evidence pack: the original behavior inventory, approved differences, comparison results, unresolved cases, and cutover rehearsal. Decide whether to expand only after another engineer and the workflow owner can explain why that evidence is sufficient. Stronger models make the experiment more attractive; clear acceptance criteria make its result useful.