One-sentence Summary

HKUST, Tencent, and HKUST(GZ) introduce DeepSearch-Evolve, a self-distillation framework that trains long-horizon web agents through iterative trajectory generation and fine-tuning in the verifiable DeepSearch-World environment with 420K multi-hop QA tasks, achieving 31.2% on BrowseComp, 61.5% on GAIA, and 93.4% on HotpotQA without external teacher models.

Key Contributions

• DeepSearch-World is a deterministic, verifiable offline environment that provides reproducible search and page-reading tools over Wikipedia, along with 420K multi-hop QA tasks and entity-level progress verification for training deep-search agents.
• A scaffold process supervision method converts teacher-scaffolded rollouts (tracking progress, evidence, failures, and recovery) into standard ReAct-format training data, injecting planning, memory, grounded reflection, and failure recovery into the student agent.
• The DeepSearch-Evolve self-distillation framework iteratively generates, filters, and fine-tunes on verified trajectories, enabling open-source agents to improve from their own tool-use experience without relying on synthetic data from stronger proprietary models.

Introduction

Large language model (LLM) agents can now plan, search, browse, and reason with tools, but enabling them to improve autonomously from their own interactions remains a critical challenge for building scalable self-evolving agents. Prior approaches either fine-tune on static positive trajectories (which quickly saturates), apply RL with sparse trajectory-level rewards (offering no step-level guidance), or attempt dense on-policy self-distillation (which requires a reliable teacher distribution that is hard to obtain in open-ended tool-use settings). The authors address these limitations by introducing DeepSearch-World, a deterministic and verifiable environment for deep search agents that provides reproducible observations and entity-level progress verification over 420K multi-hop QA tasks. Building on this environment, they propose DeepSearch-Evolve, a self-distillation framework that collects verified trajectories, converts scaffold process supervision (tracking progress, evidence, failures, and recovery) into standard ReAct-format training data, and iteratively fine-tunes the agent to internalize planning, memory maintenance, and failure recovery. This enables open-source agents to improve from verified self-generated experience without relying on stronger proprietary models.

Dataset

The authors construct a multi-hop QA dataset and a supporting offline Wikipedia corpus to serve as the deterministic DeepSearch-World environment. The data is built from the Wikipedia hyperlink graph and used for self-distillation training of an agent.

• Dataset composition

  • 420K QA instances generated from entity-level random walks on the graph, where each walk defines an H-hop reasoning chain. Entity mentions are obfuscated, so the agent must recover the target entities through search, query revision, and evidence discovery.
  • A held-out validation split, DeepSearch-Val, containing 377 high-quality instances. Each instance is independently verified by at least three out of five experts and is guaranteed to be answerable with the provided offline corpus and search tools.
  • An offline Wikipedia corpus of roughly 10 million entries, each comprising a title, a caption, and the full article text. The corpus is built by crawling the Wikipedia pages of all target entities that appear in the QA instances.

• Data sources and processing

  • The Wikipedia hyperlink graph is sampled with entity-level random walks, producing sequences of target entities.
  • Explicit entity mentions in the walk are obfuscated to create question-answer pairs that require multi-hop reasoning.
  • The union of all target entities across the 420K instances is used to crawl corresponding Wikipedia pages, from which the 10 million-entry offline corpus is formed. This ensures all required evidence is searchable and verifiable inside the environment.
  • Validation instances are filtered to only those whose evidence is fully covered by the offline corpus and retrievable with the environment’s search and visit tools; they are then manually validated by an expert pool.

• How the data is used

  • The 420K QA pool (excluding the 377 validation instances) serves as the task distribution for training. The agent interacts with the offline Wikipedia corpus to solve these tasks, generating trajectories that are later used for self-distillation.
  • DeepSearch-Val is reserved for validation and behavioral analysis, never used for trajectory generation or model training.

Method

The authors present DeepSearch-Evolve, a self-distillation framework for search agents. As shown in the framework diagram, the system comprises three key components: a verifiable tool environment named DeepSearch-World, a scaffold teacher agent for generating high-quality trajectories, and an iterative self-evolving training loop.

To support scalable self-distillation, the authors construct a deterministic offline Wikipedia environment. They sample entity-level random walks from the Wikipedia hyperlink graph to create 420,000 multi-hop QA instances by obfuscating explicit entity mentions. The environment exposes two offline tools aligned with standard web agent workflows: a search tool using BM25 retrieval to return snippets and URLs, and a visit tool that maps URLs to full article text via a SQLite index. During rollout, the environment maintains a ground-truth entity set for process-level verification. If a tool response matches an unresolved entity, it is marked as successful. Failed calls trigger staged rule-based reflection, guiding the agent from generic revision signals to specific entity names, thereby creating grounded search-fail-reflect-retry trajectories.

The scaffold teacher agent operates in three phases: PLAN, ACT, and END. As shown in the figure below, the Planning Phase initializes a structured progress state containing fields for completed subgoals, pending actions, lessons from failures, and extracted evidence. In the Action Loop, the teacher performs interaction steps where it selects tool calls and updates its state according to the equation $s_{t+1} = \mathcal{U}(s_t, a_t, o_t, r_t)$st+1​=U(st​,at​,ot​,rt​), where $r_t$rt​ is the environment-grounded reflection. Finally, in the End phase, the teacher generates a concise answer grounded in the verified working memory.

An example of this process is shown in the figure below. The teacher decomposes a complex question, alternates between search and visit tools, and updates its working memory. When a search fails, such as querying for "Baroque crater coordinates" and finding no result, the environment provides grounded reflection. The teacher then revises the query to "Rembrandt named place coordinates" and successfully recovers, ensuring the final answer is supported by verified observations.

The self-evolving training loop alternates between trajectory generation and model updates. The current model acts as the teacher to generate scaffold trajectories in the virtual environment. Verified successful trajectories undergo rejection sampling and quality filtering to remove redundancy and weak goal alignment. These are then converted into standard ReAct-format supervision. The scaffold-specific prompts are removed, and progress states and reflections are rewritten into a thinking block. For each step $t$t, the target thinking block is constructed as:

Here, $P_t$Pt​ summarizes the progress state, $R_t$Rt​ rewrites the reflection as self-correction, and $A_t$At​ preserves the action rationale.

The authors adopt evolving Supervised Fine-Tuning (SFT) for stability in long-horizon tool use. While On-Policy Self-Distillation (OPSD) matches the student to a soft teacher distribution on student-induced prefixes, fully on-policy supervision can be unreliable if student rollouts drift into low-quality states. Instead, the authors sample scaffold rollouts, filter them, and convert them to ReAct traces. The SFT objective minimizes the KL divergence between the hard target distribution and the model policy:

This approach provides dense token-level supervision using verified offline prefixes, trading full distributional matching for stable and efficient training.

Experiment

The evaluation benchmarks DeepSearch-World-9B on seven deep search and reasoning tasks, comparing it against proprietary and open-source agents under a self-evolving training setup that uses only environment-verified rollouts from its own backbone. The model achieves competitive open-source performance without relying on stronger teacher models, demonstrating large and consistent improvements over Qwen3.5-9B-Instruct across all tasks. Analysis reveals that the self-evolution benefits from diverse data pools, leads to substantially longer and more evidence-grounded tool use, and depends critically on reflection rewriting and rejection sampling to ensure trajectory quality. These findings indicate that deterministic environment verification and carefully curated self-distillation can effectively replace external supervision for training capable deep search agents.

DeepSearch-World-9B, optimized only from its own environment-verified rollouts, achieves competitive open-source performance on deep search benchmarks without relying on stronger teacher models or synthetic pipelines. It consistently and substantially outperforms its base model Qwen3.5-9B-Instruct across all tasks, with gains reflecting transferable skills in query reformulation, evidence grounding, and multi-step synthesis. The agent also sustains much longer tool-use interactions and performs more evidence-gathering actions, indicating stronger document-level reasoning. Self-distillation from environment-verified rollouts allows DeepSearch-World-9B to surpass open-source agents that use multi-agent distillation or synthetic SFT trajectories from frontier models. The model improves over Qwen3.5-9B-Instruct by +23.8 on BrowseComp, +37.6 on GAIA, and +48.1 on HotpotQA, while extending average tool-use rounds from 4.7 to 18.0 and evidence calls from 0.9 to 5.4.

Ablation reveals that rejection sampling (RS) is the dominant factor, boosting SearchQA by a large margin. Trajectory quality filtering (QF) adds a smaller improvement on its own, but when paired with RS, it yields the highest performance by discarding redundant or inconsistent answer-correct traces. The combined approach surpasses existing methods such as SDAR and Skill-SD, underscoring the effectiveness of verified trajectory filtering for self-evolving SFT. Rejection sampling provides the major gain, raising SearchQA from 46.4 to 54.9, confirming that answer-correctness verification is the key safeguard for self-distillation. Combining quality filtering with rejection sampling achieves 58.2, outperforming isolated filters and prior methods, because it removes redundant, weakly aligned, or inconsistent answer-correct traces.

The full pipeline combining reflection rewriting and state internalization achieves the highest DeepSearch-Val score of 31.9, a clear improvement over the 25.0 SFT baseline. Reflection rewriting is critical; without it, performance drops sharply to 16.7, while state internalization provides consistent additional gains. Omitting both components yields the lowest score of 14.8, underscoring their combined importance. Reflection rewriting is the dominant factor: removing it causes a large performance drop, while state internalization contributes a smaller but consistent improvement. Injecting planning, memory, and progress tracking via state internalization lifts performance by over 8 points compared to omitting it. The full pipeline improves over vanilla SFT by a substantial margin, demonstrating the value of evolving self-distillation with these components.

DeepSearch-World-9B, trained entirely via environment-verified self-distillation without external teacher models, substantially outperforms its base model and open-source agents that rely on multi-agent distillation or synthetic data, demonstrating strong transferable skills in query reformulation, evidence grounding, and multi-step reasoning with longer tool-use interactions. Ablation studies reveal that rejection sampling is the dominant factor driving gains, while trajectory quality filtering, reflection rewriting, and state internalization each contribute further consistent improvements. The full pipeline combining these components yields the highest overall performance, confirming that self-evolution with careful trajectory filtering and agentic state modeling is highly effective for deep search tasks.