AutoSynthData Generates Targeted Synthetic Data for Enterprise Agents

ServiceNow CoreAI has introduced AutoSynthData, a framework for generating synthetic training data tailored to enterprise AI agents. The system addresses the limitation of general models in navigating specific organizational environments defined by unique workflows, tool integrations, and policy constraints. AutoSynthData systematically converts agent capability gaps into targeted training tasks by leveraging the failures of a target model and the successes of a stronger teacher model. The pipeline begins by evaluating the target agent to identify weaknesses, which are distilled into sanitized specification cards. These cards guide the generation of new tasks without exposing sensitive entities or raw trajectories. Each generated task includes a system specification outlining constraints, a realistic user prompt, and a verifier to assess outcomes. Tasks are designed to be feasible within the environment, realistic to actual user requests, and sufficiently difficult to expose model limitations. The generation process operates in two phases: a target phase creates core samples, and a multiply phase produces novel variants by altering states, entities, and wording. Quality control is enforced through rigorous validation. Candidates undergo solver evaluation, positive verification to confirm correct solutions, and negative verification to reject weak criteria. Failed samples enter a critique and repair loop, where diagnosis drives targeted corrections. Batch-level reviews monitor dataset diversity and coverage, directing generation toward unexplored capability regions. This approach ensures the resulting dataset provides a balanced and effective learning signal. ServiceNow demonstrated AutoSynthData's efficacy using EnterpriseOps Gym, benchmarking performance in Hybrid and ITSM domains with the Gemma-4-26B-A4B-it model. Using Qwen3.8-27B and DeepSeek-V4.1-Flash as teachers, the system generated approximately 2,000 samples per domain. In the Hybrid domain, generation completed in 18 hours, improving mean Pass@1 by 7.2 percentage points and closing 59% of the gap with reference models. The ITSM domain saw Pass@1 increase from 18.77% to 27.18% over 66 hours. The framework supports a continuous improvement cycle, where post-training results refine subsequent data generation rounds. ServiceNow notes the system is compatible with reinforcement learning and has released the EnterpriseOps Gym dataset to support research into synthetic data for enterprise agents.