LLM Code Transformation Accuracy Benchmarks

LLM code transformation accuracy benchmarks measure tasks ranging from isolated function completion to full-repository migration. Their scores are useful only when read at the right scope. A strong result on a small, self-contained task does not establish migration readiness, and production decisions require evidence about the target languages, test-set contamination, and the pipeline around the model.

Why a single accuracy score cannot describe LLM code transformation performance

The gap between function-level performance and full-repository transformation is not marginal. Models that handle most isolated programming tasks can struggle to complete even a small fraction of complex migrations. What does each score actually measure, and how do the results relate to one another? Interpreting the scores begins with the task being evaluated.

Two sources of confusion obscure that distinction. Scope confusion treats function completion, issue resolution, and migration as interchangeable forms of coding, while score inflation can make familiar benchmark tasks look like evidence of general capability. A team choosing a model for repository-scale migration cannot establish its suitability from a function-level score, however strong that score appears.

What "scope" means in code transformation and why it drives the accuracy ceiling

Scope means the unit of code being transformed and the complexity of its dependencies. Function-level tasks are self-contained, stateless transformations that do not require an understanding of a broader codebase. Repository-level work adds cross-file dependencies, test suites, build systems, and runtime behavior. Full-stack migration extends that work to technology-stack replacement, including toolchain changes, language rewrites, and behavioral equivalence verification.

Each step introduces new kinds of failure. A correct isolated function can still break an inter-module contract when placed in a repository. Cross-language work adds dependence on training coverage for the language pair and on how much relevant context the model can process. This explains why saturation and severe difficulty coexist across benchmarks, as a bounded task can become routine while interacting changes across a codebase remain difficult.

Function-level benchmarks: what saturation looks like and why it matters for calibration

HumanEval evaluates 164 Python programming tasks through functional correctness tests. Together with MBPP, it represents a tier that frontier models have effectively solved, reducing its ability to distinguish their transformation capabilities. That does not make the benchmark useless. HumanEval can still compare smaller, open-weight, or fine-tuned models and detect regressions where isolated programming ability remains a meaningful discriminator.

Saturation also raises questions about exposure to benchmark material. GSM8K provides a cautionary parallel outside code transformation, having begun as a difficult benchmark before frontier models crossed 90%. Contamination analysis found that accuracy fell by a meaningful amount when training-set overlap was removed. LiveCodeBench tackles this exposure problem by updating its questions on a rolling basis.

For frontier comparisons, chasing small gains on saturated tasks provides less useful evidence than moving to a broader scope. Rolling updates improve contamination resistance, but freshness alone does not turn a bounded programming task into a migration test. This tier still does not have the repository context, behavioral equivalence assessment, and cross-language fidelity needed to test broader transformation work.

SWE-bench Verified: the real-software-engineering tier and its contamination problem

SWE-bench Verified evaluates 500 real GitHub issues from open-source Python repositories, requiring working patches validated by unit tests. The model must find its way through existing code as well as complete an isolated function. Anthropic's system card records Claude Opus 4.6 at 80.8% as of early 2026. That is substantial performance on a harder task, but the score cannot be interpreted independently of exposure to the evaluation material.

OpenAI's audit found that every frontier model tested could reproduce verbatim gold patches or problem-statement specifics for certain tasks. Memorization therefore complicates the interpretation of successful patches. The uncertain value of a correct answer as evidence that a model can derive the same fix in an unfamiliar repository reflects a structural vulnerability of static public benchmarks meeting models trained on increasingly broad collections of material.

OpenAI addressed that limitation in “Why SWE-bench Verified No Longer Measures Frontier Coding Capabilities,” publicly retiring a benchmark it helped popularize. The response marks a change in requirements for repository-based evaluation. Older comparisons and smaller-model evaluations are not automatically invalidated. For frontier evaluation, however, test tasks need to be kept further apart from what models have seen during training.

SWE-bench Pro: harder tasks, multiple languages, and what the score drop reveals

SWE-bench Pro tests work in repositories with harder tasks, more languages, and held-out evaluation material. The instructive comparison is the drop from a Verified score in the high 90s to roughly half that on Pro under standardized scaffolding and unseen tasks. The model has not suddenly lost general ability. The test changes both the work the model must do and how much it can draw on material it has seen before.

That gap reflects scope and contamination together, showing why percentages from different benchmarks cannot form a single accuracy ranking. Beam 1.0's 66.1% result also establishes that open-weight models can be competitive at this tier. It does not establish comparable readiness for full-stack migration.

Pro includes Python, Go, TypeScript, and JavaScript, extending beyond Verified's Python repositories. Teams working in Java, Kotlin, Rust, or C++ do not get direct evidence for their stack from this issue-resolution coverage. A harder benchmark is more informative only where its tasks remain relevant to the intended work.

Cross-language migration benchmarks: Vals Code Migration and CodeTransBenchmark

These are different skills, even when leaderboard percentages look similar. A team planning a migration from a legacy language to Java should therefore give migration evidence a different weight than a team fixing Python bugs would, while checking which language pairs the migration benchmark covers. The evaluation needs to test the kind of fidelity the project requires.

Vals reports Claude Sonnet 5.5 first at 69.83% as of October 6, 2026, followed by Gemini 4 Argon at 68.17% and GPT-6 Astra at 67.74%. The accuracy results cluster closely, but their operating costs differ substantially. Claude Sonnet 5.5 costs $75.83 per test, compared with $44.36 for GPT-6 Astra. GPT-6 Astra completes runs in 58m30s, compared with over three hours for Claude Sonnet 5.5. Rank alone therefore leaves out trade-offs in cost, time, and accuracy that matter as the amount of migration work and the tolerance for error change.

CodeTransBenchmark evaluates eight models on three datasets across 12 language pairs. It finds that models trained specifically for multilingual coding correctly translate the majority of code, while general-purpose models struggle with the target language's syntactic rules. Its analysis also identifies the influence of training-data coverage and context-window limits. A task-agnostic leaderboard does not expose those translation-specific constraints.

SWE Refactor Bench: what full-repository stack migration actually looks like

SWE Refactor Bench, released August 24, 2026, evaluates full-repository technology-stack migration. Across 520 runs of 8 frontier models, only 28 passed all evaluation stages, and no model produced an accepted solution for 13 tasks. These results cover the hardest transformation scope and leave open how well LLM-assisted migration works on other tasks. The benchmark counts work as complete only when it is finished, so its scores show more than partial progress.

Claude Opus 5 had the best overall score at 47.0/100. Language rewriting scored only 5.6, while toolchain rewriting scored 31.4, exposing a substantial difference between the two sub-tasks. Stack migration is not one uniform capability. Replacing the language presents a larger unresolved challenge than changing the toolchain in this evaluation.

The benchmark separates “migration completeness” from “behavioral correctness” as independent capability dimensions. Among runs that passed the migration audit, the majority reached near-full scores on fixed checks, but only a fraction reached 100%. Apparent completion can therefore conceal broken behavior. Conversely, tests alone do not establish that the requested migration is complete, so either dimension used in isolation can overstate success. Google's finding that simple prompting is insufficient beyond the simplest migrations is consistent with this demanding full-pass criterion.

Domain-specific fine-tuning and rule synthesis: a different lever for transformation accuracy

Research on post-quantum cryptographic migration shows how adapting a model to a specific domain can change its accuracy. A fine-tuned model achieved a dynamic functional correctness rate of 92.5%, substantially outperforming its zero-shot baseline on the same task. The comparison is specific to that domain. It demonstrates that a general-purpose benchmark score need not represent the attainable performance of an adapted model on a constrained transformation.

Code Transformation Rule Synthesis using LLMs examines another route, generating reusable transformation rules. Such rules can be audited, versioned, and applied across a codebase without repeating model inference for each match. This changes both reviewability and cost at scale. For repetitive transformations, the relevant system includes the model's initial response, the generated rule, and its application.

The study also finds limits to this approach. Smaller open-weight models remain effective for simpler, localized changes but struggle with complex migration scenarios even when generating rules. Adaptation changes what a standard-condition score predicts without removing the scope hierarchy.

How agentic pipelines and automated feedback change what benchmark scores predict

CodeTransBenchmark finds that iterative translation repair through automated feedback significantly improves accuracy. The same underlying model can therefore produce different outcomes in a single-shot evaluation and a repair loop. Production accuracy depends on that surrounding process. A benchmark result remains evidence of capability, but only under the conditions that produced it.

Google's migration toolkit uses LLM generation along with syntax-tree-based discovery, automated test validation, and review by people at commit boundaries. About 87% of AI-generated code in its migration to a new testing framework version was committed without changes. That figure reflects the pipeline's filtering rather than an isolated measure of raw model output quality. It cannot be compared directly with a benchmark pass rate that uses different acceptance conditions.

Scaffolding can amplify capability or constrain it. Test coverage and human review cadence also shape the reliability of the resulting transformation, so two organizations using the same model can observe different accuracy without either measurement being wrong. Evaluations embedded in CI/CD make benchmark results an input to pipeline design.

How to read any code transformation accuracy number: a practical interpretation framework

A team interpreting a published score should establish its scope, contamination resistance, language coverage, and execution conditions. Scope comes first. HumanEval and MBPP remain directional for smaller models performing isolated rewrites or simple updates, while LiveCodeBench offers rolling evaluation material. SWE-bench Pro's held-out set is the relevant reference for repository issue resolution, whereas Verified requires the contamination caveat. Vals and CodeTransBenchmark address translation, and SWE Refactor Bench addresses full-stack replacement.

For frontier comparisons, teams should prefer held-out or rolling test sets over exposed static tasks. Language coverage needs a separate check. Evidence from Python, Go, TypeScript, and JavaScript issue-resolution tasks cannot establish performance on a different stack, and multilingual translation evidence should be matched to the actual language pair. When public tests do not cover the project's work, teams need samples of their own code that reflect that work.

Teams should ask whether the score came from raw output or an agentic scaffold, and whether that scaffold is reproducible in their environment. Domain-specific fine-tuning and rule synthesis deserve consideration where the work is constrained or repetitive. Internal evaluations should use a fixed sample of the target codebase and score migration completeness separately from behavioral correctness. The resulting evidence answers the deployment question that a leaderboard alone cannot answer, which is whether this transformation process can complete this team's work correctly.