ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration

Hugging Face has introduced ScarfBench, a new benchmark designed to evaluate AI agents on the task of migrating enterprise Java codebases between frameworks. The benchmark simulates real-world migration challenges, such as converting Spring Boot applications to Quarkus or modernizing legacy Java EE projects. ScarfBench includes a suite of microservices and migration tasks that test an agent’s ability to understand code semantics, handle dependencies, and maintain functionality across framework changes. For developers building AI workflows, this benchmark offers a standardized way to measure agent performance on complex, multi-step code transformation tasks—beyond simple code generation. The Hugging Face team reports that even state-of-the-art agents currently struggle with these migrations, highlighting gaps in reasoning about enterprise-scale codebases. This research underscores the need for agents that can navigate context-heavy, long-horizon tasks in real-world engineering environments.