Lianghong Guo Yanlin Wang Caihua Li Pengyu Yang Jiachi Chen Wei Tao Yingtian Zou Duyu Tang Zibin Zheng

Abstract

Constructing large-scale datasets for the GitHub issue resolution task iscrucial for both training and evaluating the software engineering capabilitiesof Large Language Models (LLMs). However, the traditional process for creatingsuch benchmarks is notoriously challenging and labor-intensive, particularly inthe stages of setting up evaluation environments, grading test outcomes, andvalidating task instances. In this paper, we propose SWE-Factory, an automatedpipeline designed to address these challenges. To tackle these issues, ourpipeline integrates three core automated components. First, we introduceSWE-Builder, a multi-agent system that automates evaluation environmentconstruction, which employs four specialized agents that work in acollaborative, iterative loop and leverages an environment memory pool toenhance efficiency. Second, we introduce a standardized, exit-code-basedgrading method that eliminates the need for manually writing custom parsers.Finally, we automate the fail2pass validation process using these reliable exitcode signals. Experiments on 671 issues across four programming languages showthat our pipeline can effectively construct valid task instances; for example,with GPT-4.1-mini, our SWE-Builder constructs 269 valid instances at 0.045 perinstance, while with Gemini-2.5-flash, it achieves comparable performance atthe lowest cost of 0.024 per instance. We also demonstrate that ourexit-code-based grading achieves 100% accuracy compared to manual inspection,and our automated fail2pass validation reaches a precision of 0.92 and a recallof 1.00. We hope our automated pipeline will accelerate the collection oflarge-scale, high-quality GitHub issue resolution datasets for both trainingand evaluation. Our code and datasets are released athttps://github.com/DeepSoftwareAnalytics/swe-factory.