Kaggle ScarfBench: Can Agents Migrate Enterprise Java Apps?

Enterprise Java modernization fails less often on “does the code look like the target framework” than on what happens next: it compiles, then will not boot—or boots with drifted routes and business behavior. IBM Research’s ScarfBench (Self-Contained Application Refactoring Benchmark) turns that gap into a public yardstick: can coding agents migrate apps across Jakarta EE, Quarkus, and Spring while preserving runnable behavior?

IBM introduced the benchmark earlier this year. The newer move is a living leaderboard with Kaggle: Kaggle advised on evaluation design, independently reproduced runs on its platform while sponsoring compute, and will keep adding models. The suite has 204 directed migration tasks from 34 application families (focused single-layer examples plus whole apps). A candidate must compile, deploy in the target runtime, and pass behavioral tests—compilation alone does not count.

Across ten model-and-harness combinations, Gemini 4 Argon with Antigravity led the primary smoke metric at about 75.2%, followed by Claude Opus 5.5 with Claude Code at about 70.5% and Claude Opus 5 with Claude Code at about 69.8%. The nuance matters: smoke awards partial credit and weights larger suites more, which favors Argon; on full test-pass rate, Opus 5.5 edges ahead at about 72.1% versus Argon’s about 71.1%. Compile success is generally high; runtime fidelity—lost routes, HTTP drift, leftover source-framework wiring—still separates winners from near-misses.

For tool builders and enterprise architects, ScarfBench’s value is not another screenshot of a ranking. It pulls “modernization agents” out of demo videos and into reproducible acceptance tests. A top smoke score near 75% shows frontier stacks can chew through a real share of migrations—and that roughly a quarter of the hard cases remain. Public submissions are not fully open at launch, but they are on the roadmap. Until then, the Kaggle board is already enough to make the next Java-agent pitch lean less on slogans and more on behavioral equivalence.