UM Understanding & modernisation
Legacy estates (Java, .NET, C/C++, COBOL alike), migration, code comprehension.
From the issues
Legacy PHP: the best modernisation tool was the one that did nothing
Not by default. When five LLMs upgraded 100 WordPress files to PHP 8.3, 83 files already passed every behavioural check untouched and the best model left only 59 passing. Each model broke 32 to 51 working files, against 7 for the rule-based tool Rector. Demand a do-nothing baseline beside any migration score.
"86.6% on COBOL": the number the post doesn't let you check
How good is AI at COBOL migration? OpenAI says GPT-6.1 Sol scores 86.6% at low effort, up from 5.4% for the previous model, and 97.9% at higher effort. It's self-reported, and the post doesn't say how many programs were tested or whether copybooks and job control were in scope. Ask for the estate, not the file.
When the Model Retires: your AI stack is already legacy
Most teams move off a retired AI model after it breaks, not before. A study of 22,555 commits across 17,703 GitHub repos estimates 82% of migrations landed after the shutdown date, and 94% of the apps had the model ID hard-coded. Put model IDs in config, with an owner and a retirement date.
Numbers on this track
| Number | What it measures | Source | Checked | Track · issue |
|---|---|---|---|---|
| 82% | Estimated share of LLM migrations made after the model's shutdown date (22,555 commits, 17,703 GitHub repos) | Kim, arXiv:2609.31288 | ✓verified at source | UMUnderstanding & modernisation Issue #01 |
| 94% | LLM apps with the model ID hard-coded | Kim, arXiv:2609.31288 | ✓verified at source | UMUnderstanding & modernisation Issue #01 |
| 89% vs 13% | Post-shutdown migrations under Anthropic's 60 to 114-day notices vs OpenAI's one-year notice | Kim, arXiv:2609.31288 | ✓verified at source | UMUnderstanding & modernisation Issue #01 |
| 8% | LLM migrations that switched model provider | Kim, arXiv:2609.31288 | ✓verified at source | UMUnderstanding & modernisation Issue #01 |
| 26.4% / 20.4% | Best success rate on Linux-target and Windows-target cross-OS migration tasks (140 tasks) | Jin et al., OS-Bench, arXiv:2610.19368 | ✓verified at source | UMUnderstanding & modernisation Issue #03 |
| 86.6% | GPT-6.1 Sol's accuracy at low reasoning effort on OpenAI's COBOL migration evaluation (previous model 5.4%; 97.9% at higher effort; 61% of the API cost) | OpenAI, low-resource programming languages post | self-reported | UMUnderstanding & modernisation Issue #03 |
| 83 vs 59 | WordPress files passing every hidden check untouched, against the best LLM after upgrading to PHP 8.3 (100 files, 630 checks; abstract) | Mahmud et al., arXiv:2610.40269 | ✓verified at source | UMUnderstanding & modernisation Issue #04 |
| 32 to 51 vs 7 | Working files broken by each LLM, against the rule-based tool Rector (abstract) | Mahmud et al., arXiv:2610.40269 | ✓verified at source | UMUnderstanding & modernisation Issue #04 |
| 7% | Share of 511 findings (37) that needed a change under PHP 8.3 (abstract) | Mahmud et al., arXiv:2610.40269 | ✓verified at source | UMUnderstanding & modernisation Issue #04 |
Papers on this track
- When the Model Retires: An Empirical Study of LLM Migration in Open-Source Applications arXiv:2609.31288
- OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration arXiv:2610.19368
- LLM upgrades of legacy WordPress PHP files, with do-nothing and Rector baselines arXiv:2610.40269
Other tracks
ISIntent & specs CKContext & knowledge ABAgents that build VTVerification & trust POPeople & operating model
Last updated . Reuse with credit under CC BY 4.0: “The Dabbawala Protocol, thedabbawalaprotocol.com”.