# When Modernization Breaks Working Code: A Single-File Study of LLMs on Legacy PHP

DevFeed: [When Modernization Breaks Working Code: A Single-File Study of LLMs on Legacy PHP](<https://devfeed.tech/articles/when-modernization-breaks-working-code-a-single-file-study-of-llms-on-legacy-php-107408.md>)

Original publisher: [Read original article](<https://arxiv.org/abs/2610.40269>)

Author: Afridi Mahmud, Abid Hasan Khondaker, Sadif Ahmed, Md Nafiu Rahman, Rifat Shahriyar

Published: 2026-10-30T04:00:00Z

Content type: article

Language: en

Sources: [arXiv: Software Engineering](<https://devfeed.tech/sources/arxiv-software-engineering.md>)

Topics: [Legacy Modernization](<https://devfeed.tech/topics/legacy-modernization.md>), [Refactoring](<https://devfeed.tech/topics/refactoring.md>), [legacy systems](<https://devfeed.tech/topics/legacy-systems.md>), [test](<https://devfeed.tech/topics/test.md>)

Tags: [abstract](<https://devfeed.tech/tags/abstract.md>), [all](<https://devfeed.tech/tags/all.md>), [arxiv](<https://devfeed.tech/tags/arxiv.md>), [avoid](<https://devfeed.tech/tags/avoid.md>), [based](<https://devfeed.tech/tags/based.md>), [behavior](<https://devfeed.tech/tags/behavior.md>), [best](<https://devfeed.tech/tags/best.md>), [better](<https://devfeed.tech/tags/better.md>), [but](<https://devfeed.tech/tags/but.md>), [change](<https://devfeed.tech/tags/change.md>), [code](<https://devfeed.tech/tags/code.md>), [compare](<https://devfeed.tech/tags/compare.md>), [context](<https://devfeed.tech/tags/context.md>), [cs-se](<https://devfeed.tech/tags/cs-se.md>), [easy](<https://devfeed.tech/tags/easy.md>), [file](<https://devfeed.tech/tags/file.md>), [legacy](<https://devfeed.tech/tags/legacy.md>), [llms](<https://devfeed.tech/tags/llms.md>), [modernization](<https://devfeed.tech/tags/modernization.md>), [php](<https://devfeed.tech/tags/php.md>), [study](<https://devfeed.tech/tags/study.md>)

## AI overview

In a study of 500 WordPress 4.0 PHP files modernized to PHP 8.3 without project context or execution feedback, LLMs often damaged working code. The untouched originals had 83 files passing all 630 behavioral checks; the best model's output had 59. Each model broke 32-51 working files and repaired at most 10, while Rector broke 7 and repaired 6. Only 37 of 511 modernization findings required a PHP 8.3 change. Even among compatible files needing no change, LLMs broke 36-60%, compared with Rector's 8%. Many regressions prevented files from loading, often due to PHP-rejected type declarations. The authors recommend evaluating modernization against an unchanged baseline and reporting modernization separately from behavioral preservation.

## Source excerpt

arXiv:2610.40269v1 Announce Type: new Abstract: Pasting a legacy file into an LLM and asking it to "modernize this" is an easy shortcut. But is the result better than leaving the code alone? In this paper, we study this single-file setting with five LLMs, each asked to modernize 100 WordPress 4.0 PHP files to PHP 8.3 while preserving behavior, with no project context or execution feedback. We evaluate every output with 630 hidden behavioral checks and 511 Rector-detected modernization findings. We compare against two baselines: the unchanged originals and the rule-based tool Rector. Often, the answer is no. Untouched, 83 files already pass every check; the best LLM leaves only 59. Each model breaks 32-51 working files while repairing at most 10, whereas Rector breaks 7 and repairs 6. Most of this damage is unnecessary. Only 37 findings (7 percent) require a change under PHP 8.3, yet in compatible files with no required change, the LLMs break 36-60 percent (Rector: 8 percent). The model with the highest modernization score is no more compatible than its closest competitors. Most regressed checks come from files that no longer load at all, often because of type declarations PHP itself rejects. These are errors that need no project context to avoid. Evaluations of LLM-based modernization should therefore measure against a no-change baseline and report modernization and behavioral preservation separately.