RefactorPlatform

An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents

1Pi School, 2Translated

EMNLP 2026 · System Demonstrations

*Equal contribution.

Coding tool step: the agent plugin, a free-text model field, and the provider's searchable model catalogue
(a) Configure
The same task's terminal, replayed from the captured PTY bytes with workspace paths redacted
(b) Watch it run
Run detail: agent steps on the left, live terminal and diff in the centre, evaluation checks and token accounting on the right
(c) Inspect the diff
Settings, Evaluation and metrics: the verdict rule over three gating stages, each with its failure reason, what it records, which plugin provides it and how many options it accepts, above the metrics recorded on every task
(d) Verify the result

RefactorPlatform walks an operator from configuring a run to verifying its result, live.

Abstract

An open-source evaluation harness for repository-scale refactoring agents. It holds the execution environment fixed and varies one design axis at a time — model backbone, execution regime (baseline, retrieval-augmented, multi-agent), retrieval chunking strategy, and prompt specificity — with per-task isolated workspaces, live terminal streaming, token and diff telemetry, AST-based verification for Python, and compilation, test and RefactoringMiner gates for Java.

Results

86% Best pass rate
(qwen3.6-flash, S2 AST retrieval)
+25–30pp AST-aware vs. naive chunking
$0.13 Cost per successful refactor
Cross-model pass rate on RefactorBench (100 tasks), descriptive prompts.
Model S1 Baseline S2 + AST Retrieval
qwen3.6-flash 73% 86%
minimax-m3 75% 81%
kimi-k2.6 71% 78%
deepseek-v4-pro 77% 89%

A single lean retrieval-augmented agent (86%) also outperforms the one orchestrated multi-agent topology evaluated (77%, S3, Copilot sub-agents), and retrieval's accuracy gains absorb its token overhead: cost per successful refactoring shifts by no more than $0.01 across all four models. See the paper for the full ablation, cost-efficiency, and Java (SWE-Refactor) results.

Structural retrieval

AST-aware chunking sits alone on the purity/integrity frontier that naive fixed-line windows only approach at much larger window sizes.

Integrity against purity for each chunking strategy: fixed line windows trace a steep trade-off from 20-line windows at 30 percent purity and 63 percent integrity to 240-line windows at 10 percent purity and 98 percent integrity, while AST chunks sit alone at 93 percent purity and 92 percent integrity

Prompt sensitivity

AST-chunk retrieval leads at every prompt specificity level; naive line-window retrieval trails even the no-retrieval baseline.

RefactorBench pass counts by prompt mode: S1 agent 48/64/73, retrieval over line windows 31/44/57, retrieval over AST chunks 56/74/86

Cross-model gains

AST-aware retrieval improves pass rate for every model family evaluated, not just the flagship backbone.

RefactorBench descriptive prompt per model: deepseek-v4-pro 77 to 89, qwen3.6-flash 73 to 86, minimax-m3 75 to 81, kimi-k2.6 71 to 78

SWE-Refactor verification

Java is held to a triple gate: code extraction, structural detection, and a green build and test suite, applied in sequence.

SWE-Refactor funnel: 159 of 177 tasks produced code, 105 were detected as the requested refactoring, 50 kept the build and tests green, 43 satisfied all three

BibTeX

@inproceedings{refactorplatform2026,
  title     = {RefactorPlatform: An Open-Source Harness for Controlled Evaluation of Repository-Scale Refactoring Agents},
  author    = {Ben Amor, Aziz and Mali, Drish and Acharya, Mann and Iyer, Vijayasri and Bratières, Sébastien},
  booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  publisher = {Association for Computational Linguistics},
  year      = {2026}
}