AI legacy migration: seven parity gates

An AI-assisted legacy migration is ready for a pilot when it has a small functional boundary, reproducible inputs and outputs, business rules converted into tests, known dependencies, reconcilable data, and a rehearsed rollback path. On October 19, 2026, OpenAI published two evaluations for languages with less training code: MultiLangBench measures building a SQLite-compatible engine in 20 languages, while a separate evaluation measures COBOL migrations into five languages with hidden tests and byte-for-byte parity. The news was one Lima calendar day old when consulted on October 20. The results show a capability jump inside those environments; they do not show that a model automatically understands a company's jobs, files, compilers, calendars, controls, or operating agreements. The benchmark can therefore justify a controlled experiment, not approve a production replacement.

The right unit is neither the whole repository nor a count of converted lines. It is a business capability with an owner, an observable contract, and a bounded blast radius: for example, calculating a settlement, validating a batch, or producing a regulatory file. The model may propose code, tests, and explanations, but the team retains authority over the contract and acceptance. If nobody can describe what must remain identical and what may change, there is no evaluable migration yet—only a rewrite.

MultiLangBench gives the model SQLite 3.40.1 documentation, removes SQLite from the environment, blocks internet access, and requires an executable in the target language. Each case prepares an initial database, sends a query, and uses real SQLite to inspect the resulting file. The repository converts roughly 24,000 cases from the TCL suite and recommends a deterministic subset of 272 for ordinary evaluations. This avoids grading style or textual similarity: it asks whether the artifact changed state correctly. OpenAI reports that GPT-6 Astra reached an 87.4% average at maximum effort across 20 languages, compared with 26.8% for GPT-5.6 Sol in that setting.

The COBOL evaluation asks a different question: whether an existing program can be translated into C++, Java, Kotlin, Python, or Rust without changing observable outputs. OpenAI considers a migration successful only when it passes every hidden test for a case with byte-for-byte agreement, and reports 86.6% for GPT-6.1 Sol at low effort and 97.9% at higher effort. That is an important signal because it penalizes translations that compile yet change results. It also has limits: the announcement does not release this second suite, a near-saturated score may reflect its coverage, and observable equivalence alone does not cover performance, security, operations, or external dependencies.

The transferable lesson is not to copy the percentage. It is to copy the shape of the exam: a shared interface, an independent oracle, exact outcomes, hidden cases, and visible cost. A private pilot needs the same discipline with synthetic or de-identified data, plus failures the model did not see during iteration. If the same agent writes the translation, generates expectations, and explains why it passed, there is no independent verification.

Gate 1, executable boundary. Select a capability that can be invoked repeatedly and enumerate everything crossing its edge: called programs, copybooks, files, tables, queues, environment variables, scheduler, encoding, time zone, compiler, and human operators. Record versions and ownership. The gate fails if the program can only be tested inside a larger batch, if a dependency appears only through tribal knowledge, or if the modern path requires changing the consumer contract at the same time. AI can help trace calls, but an accountable person must confirm the map.

Gate 2, behavior inventory. Capture a golden set before translating: representative inputs, exact outputs, return codes, state effects, messages, record order, and expected failures. Add edges that production knows and documentation omits: empty files, duplicate records, negative amounts, month-end, year rollover, invalid characters, retries, and interrupted processing. Separate development fixtures from a holdout the agent cannot use. The gate passes when the current system can reproduce the oracle and the team can explain every accepted difference.

Gate 3, language semantics. In COBOL, a decimal expression may discard excess digits unless rounding is explicitly requested; changing that detail can alter money without throwing an exception. Review PIC, sign, scale, overflow, figurative values, comparison, collating sequence, encoding, dates, spaces, indexed files, and the behavior of the compiler in use. Do not replace an unusual rule with the target language's convention merely because it looks cleaner. Every intentional difference needs a business decision, a test, and a strategy for historical data.

Gate 4, data parity. Run the old and candidate paths against the same snapshot, in shadow mode and without irreversible dual writes. Reconcile counts, control totals, hashes, keys, rejects, and side effects; then drill down to the record that explains a difference. Design idempotency and recovery for retries. An identical final total can hide two offsetting errors, while an ordering difference may be irrelevant or may break a consumer. The policy must state what requires byte-for-byte agreement, what allows semantic equivalence, and who resolves exceptions.

Gate 5, operational equivalence. Measure the full day, not a demo: batch duration, CPU and memory, closing windows, concurrency, backpressure, observability, recovery, cost, and support load. Force timeouts, partial files, dependency outages, full disks, and restarts. Verify that alerts and runbooks lead to a concrete action. An implementation may produce every correct output and still be unfit if it misses the overnight window, multiplies cost, cannot identify the failed record, or turns automatic recovery into manual work.

Gate 6, AI control. Fix the starting commit, allowed files, available documentation, tools, network, time, and expected artifact. Use synthetic or de-identified data; do not expose credentials or production extracts for convenience. Preserve prompts, model, configuration, diffs, tests, and reviews as provenance. Separate authorship from acceptance: the agent does not modify the oracle, grant its own exception, or approve its patch. Mandatory human review should focus on contracts, semantics, concurrency, security, and code beyond test coverage.

Gate 7, reversibility. Deploy by capability, cohort, or batch with parallel observation. Keep the old system available until representative cycles, closing periods, and exceptions have been covered; define which state can be replicated both ways and which requires freezing the decision. Rehearse the return under pressure rather than merely documenting it. Measure return time, maximum work loss, follow-up reconciliation, and who has stop authority. Retiring the legacy system requires accumulated evidence and a plan to preserve auditability, not one week without visible incidents.

Suppose a hypothetical COBOL program receives transactions every night, applies fees, and produces a settlement file plus a rejection report. The team wants to migrate it to Java. It does not hand the entire repository to the model: it encloses one operating date, one product type, and one input format. The inventory reveals two copybooks, a rate table, a holiday calendar, and a job that retries rejected records. That map prevents the team from declaring success merely because the main calculation compiles.

The golden set preserves ordinary days, month-end, a holiday, duplicates, a missing rate, an amount with extra decimals, and a truncated file. The oracle requires identical bytes for files consumed by third parties, field-level equivalence for redesigned logs, and the same retry effects. The agent proposes the translation and additional tests; a separate harness runs both versions against identical snapshots. A one-cent discrepancy reveals that Java rounded where COBOL truncated. Product and finance decide to preserve the current rule during migration and open a later, explicit, separate change.

For two closing cycles, the Java path runs in shadow and reconciles every batch. It then processes a reversible cohort while COBOL retains authority for the rest. The team forces a rate-table outage, restart, and rollback; it confirms that payments are not duplicated and that support can locate the responsible record. This example describes neither a Wasyra customer nor a Wasyra result. It illustrates why value lies not in converting more lines, but in reducing uncertainty without combining modernization, functional change, and operational retirement into one bet.

Start when there is a small but valuable boundary, the current system can still produce the oracle, specialists are available to resolve ambiguity, and the organization will fund parallel execution. Wait if you expect the model to discover every rule alone, if data cannot leave production under appropriate controls, if behavior has no owner, or if the only release strategy is to switch the legacy system off. In those cases, invest first in observability, fixtures, executable documentation, and decoupling.

This week's advance changes the cost of exploring previously underserved languages and shows that behavior-based tests can guide a model better than a superficial code review. It does not remove the central work of modernization: deciding the contract, building an independent oracle, and controlling change. Use the seven gates as an investment contract. If the pilot leaves traceable evidence at each one, you can expand one unit at a time; if it fails, you will have bought bounded learning instead of an irreversible rewrite.