Experts tell The Verge AI is still far from replacing COBOL

Replacing COBOL on real production systems is still far from what AI tools can do. That is what four computer science professors told The Verge. OpenAI had said this month that its new model GPT-6.1 Sol made large gains on legacy code migration. OpenAI's spokesperson Oscar Haines told the outlet by email that the task "is still far from a solved problem".

The story matters because of what COBOL still runs. IBM says COBOL is the base of over 40 percent of online banking systems. It supports 80 percent of in-person credit card payments. It handles 95 percent of ATM transactions. The Verge cites one rough estimate that COBOL's inefficiencies cost the United States around $105 billion a year.

The vendor claim

OpenAI said in a blog post that GPT-6.1 Sol reached 87 percent behavior preservation accuracy at low reasoning effort. That score climbs to 98 percent at high reasoning. The older models scored much lower on the same test. GPT-6 Astra reached 86 percent at low reasoning. GPT-6 Sol reached only 21 percent. GPT-5.6 Sol reached only 5 percent. The tests covered migration from COBOL to C++, Java, Kotlin, Python and Rust.

Anthropic has made a similar pitch. The Verge reports that Anthropic published a "code modernization playbook" in February. The post pitched Claude Code as a way to migrate COBOL into Java or Python. Anthropic gave no benchmark. It said "the economics of COBOL modernization have shifted". IBM's share price fell after that post. Anthropic did not provide comment to The Verge in time for publication.

What the experts told The Verge

Vadim Zaytsev is an associate professor at the University of Twente in the Netherlands. He told The Verge that "rewriting code is not the same as replacing a system". He said the largest system he had seen was 250 million lines of COBOL. The code is only one part. The system also has data, integrations and user expectations.

Arie van Deursen is a software engineering professor at Delft University of Technology. He called the OpenAI blog post "superficial". He said the OpenAI benchmark tested the models on simple, well-defined tasks. Those systems were similar to the ones in the training data. He said real migration is "lightyears" away from that test.

Alastair Donaldson is a programming languages professor at Imperial College London. He told The Verge the main problem is not writing new code. The main problem is understanding how the old system works. He said very few programmers can read COBOL today. He added that "nobody in their right mind would use COBOL for a new development project".

Jacopo Mauro is a professor at the University of Southern Denmark. He said formal verification could check an AI translation. He added that applying it to legacy code would take "a huge amount of work". Until then, Mauro said, "human verification remains essential".

What a buyer should take from this

A benchmark is a claim and a system is a thing. The OpenAI tests ran on cases where passing tests could score the answer. A real COBOL system has code written over decades. It has data formats, outside connections and small differences in behavior that end users expect.

Zaytsev's last sentence puts the risk clearly. He asked what happens if a magic AI wand handles 99 percent of your code. He said that remaining 1 percent can be a million lines, spread all over the place. A buyer of software work should read a vendor's COBOL benchmark as a floor on hard tasks. It is never a ceiling on real ones.

Source

• Primary source: The Verge, "Can AI help solve the COBOL code crisis?", by Robert Hart.