One-sentence Summary

Tsinghua University, Zhejiang University, et al. propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into natural-language hindsight skills and distills their behavioral effect into a dense token-level supervision signal jointly optimized with outcome-based reinforcement learning, thereby bridging the supervision gap and significantly improving performance and sample efficiency in agentic tasks.

Key Contributions

• SEED is a self-evolving on-policy distillation framework that transforms completed on-policy trajectories into hindsight skills and distills their behavioral guidance back into the policy, allowing decision-making and skill analysis to co-evolve during agentic reinforcement learning.
• A policy-synchronized hindsight on-policy distillation mechanism converts skill-induced log-probability shifts on sampled actions into dense token-level supervision and jointly optimizes this signal with outcome-based RL.
• Experiments on embodied interaction, web navigation, search-based QA, and visual reasoning and planning benchmarks show consistent improvements in task performance, sample efficiency, and robustness over representative baselines.

Introduction

Large language model agents increasingly operate in multi-turn settings where they must reason, act, use tools, and incorporate environmental feedback over long horizons. Reinforcement learning has become a key post-training paradigm for these agents because it directly optimizes policies against task-level outcomes. However, outcome-based RL provides only sparse, trajectory-level rewards: it signals whether an episode succeeded but cannot identify which intermediate decisions, tool calls, or observations were effective. This leaves a supervision gap between coarse episode outcomes and fine-grained token-level policy learning. Prior hindsight learning methods attempt to exploit completed trajectories for richer feedback, but they typically treat hindsight as static experience, external memory, or fixed inference-time prompts, which cannot adapt as the policy improves and encounters new states and failure modes.

The authors introduce SEED (SElf-Evolving On-Policy Distillation), a framework that converts completed on-policy trajectories into natural-language hindsight skills and distills their behavioral guidance back into the policy through a self-evolving training loop. During RL, the same model collects trajectories and analyzes them to extract reusable workflows, decisive observations, and failure-avoidance rules. Because the policy and the analyzer share the same checkpoint, both decision-making and hindsight supervision co-evolve. SEED then uses the extracted skills to rescore sampled actions under ordinary and skill-augmented contexts, producing a dense token-level on-policy distillation signal that is jointly optimized with the outcome-based RL objective. This allows the policy to internalize reusable guidance without external memory or extra inference-time prompts. Evaluations across embodied interaction, web navigation, search-based QA, and visual reasoning show consistent improvements in task performance, sample efficiency, and robustness.

Method

The authors present SEED, a self-evolving on-policy distillation (OPD) framework designed for agentic reinforcement learning. SEED is motivated by the observation that completed agent trajectories contain rich hindsight information. Even when rewards are sparse, a full trajectory often reveals useful behavioral patterns, failure causes, and reusable strategies that are not directly available at intermediate decision steps. SEED converts such hindsight information into hindsight skills and distills their behavioral effect back into the ordinary policy.

As shown in the figure below, SEED consists of two main training stages. First, hindsight-skill supervised fine-tuning (SFT) equips a single policy model to analyze completed trajectories. Second, the current policy snapshot both collects on-policy trajectories and analyzes them into hindsight skills. The same sampled actions are then re-scored under ordinary and skill-augmented contexts to construct a token-level OPD signal, which is optimized jointly with RL.

Problem Formulation The authors formulate long-horizon agentic tasks as partially observable Markov decision processes $(\mathcal{S}, \mathcal{A}, \mathcal{O}, \mathcal{T}, \Omega, \mathcal{R}, \gamma)$(S,A,O,T,Ω,R,γ). At timestep $t$t, the agent receives an observation $o_t \in \mathcal{O}$ot​∈O and maintains an interaction history $h_t = (o_0, a_0, o_1, a_1, \dots, o_t)$ht​=(o0​,a0​,o1​,a1​,…,ot​). The policy generates the next action according to $a_t \sim \pi_\theta(\cdot \mid h_t)$at​∼πθ​(⋅∣ht​). A completed trajectory is denoted by $\tau = \{(o_t, a_t, r_t)\}_{t=0}^{T-1}$τ={(ot​,at​,rt​)}t=0T−1​, with an episode-level outcome $R(\tau) \in \mathbb{R}$R(τ)∈R. In many agentic environments, $R(\tau)$R(τ) is sparse and becomes available only after task completion. The standard reinforcement learning objective is $J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[R(\tau)]$J(θ)=Eτ∼πθ​​[R(τ)]. SEED bridges the granularity gap between trajectory-level feedback and token-level decisions by deriving an auxiliary training-time token-level signal from hindsight skills extracted after trajectory completion.

Hindsight Skill Supervised Fine-Tuning The first stage initializes the policy with the ability to analyze full interaction histories and express reusable behavioral guidance as natural-language hindsight skills. The authors first collect an offline pool of trajectories using a base policy without skill augmentation. For a set of training tasks $\mathcal{Q}_{\text{sft}} = \{q_j\}_{j=1}^M$Qsft​={qj​}j=1M​, they run $K_0$K0​ rollouts with the base policy $\pi_{\theta_{\text{base}}}$πθbase​​ to form the offline trajectory pool $\mathcal{B}$B. Each trajectory contains the task description, observations, actions, rewards, and final outcome, collected without any hindsight skill in the decision context.

Given a completed trajectory $\tau$τ, an external analyzer $A_{\text{ext}}$Aext​ produces a hindsight-skill annotation $s_\tau = A_{\text{ext}}(\tau)$sτ​=Aext​(τ). For a successful trajectory, $s_\tau$sτ​ captures reusable strategies or workflows, while for a failed trajectory, it encodes corrective or avoidance guidance. The annotation is retained only if it is correctly formatted, forming the accepted SFT set $\mathcal{D}_{\text{sft}}$Dsft​. The policy model is then fine-tuned to predict the hindsight skill from the completed trajectory using the standard negative log-likelihood objective: $\mathcal{L}_{\text{sft}}(\theta) = - \mathbb{E}_{(x_\tau, s_\tau) \sim \mathcal{D}_{\text{sft}}} \left[ \sum_{\ell=1}^{|s_\tau|} \log \pi_\theta(s_{\tau, \ell} \mid x_\tau, s_{\tau, < \ell}) \right]$Lsft​(θ)=−E(xτ​,sτ​)∼Dsft​​[∑ℓ=1∣sτ​∣​logπθ​(sτ,ℓ​∣xτ​,sτ,<ℓ​)] The resulting checkpoint $\theta_{\text{sft}}$θsft​ initializes the later RL policy. During RL, the trajectory analyzer is instantiated directly from the current policy checkpoint, allowing the same model to act in the environment and generate hindsight skills without a separately trained analyzer.

Self-Evolving On-Policy Distillation The second stage performs agentic RL with an additional token-level OPD signal. At the start of each update, SEED freezes the current policy as $\pi_{\theta_{\text{old}}}$πθold​​. This snapshot collects trajectories and parameterizes the analyzer that extracts their hindsight skills. A trainable policy $\pi_\theta$πθ​, initialized from $\pi_{\theta_{\text{old}}}$πθold​​, is then optimized with both the environment-driven RL and OPD objectives.

For each task prompt $q$q, SEED samples a group of $N$N trajectories using the frozen policy: $\mathcal{G}_q = \{\tau_q^{(1)}, \tau_q^{(2)}, \dots, \tau_q^{(N)}\}$Gq​={τq(1)​,τq(2)​,…,τq(N)​}. The analyzer agent, instantiated from the same policy snapshot $\pi_{\theta_{\text{old}}}$πθold​​, analyzes each completed trajectory and generates a hindsight skill $s_q^{(n)} = A_{\theta_{\text{old}}}(x_{\tau_q^{(n)}})$sq(n)​=Aθold​​(xτq(n)​​). This shared parameterization creates the self-evolving loop, where refreshing $\theta_{\text{old}}$θold​ changes both the trajectories encountered by the actor and the model capability used for skill analysis.

To construct the distillation objective, SEED keeps the original on-policy actions fixed and re-scores them under a skill-augmented context. Let $H$H be a deterministic context augmentation function. At timestep $t$t in trajectory $\tau_q^{(n)}$τq(n)​, the skill-augmented history is $\tilde{h}_{q,n,t} = H(h_{q,n,t}, s_q^{(n)})$h~q,n,t​=H(hq,n,t​,sq(n)​). The same policy computes two token-level log-probabilities on the same sampled action tokens. The skill-conditioned teacher branch log-probability is obtained by re-scoring the sampled action under the skill-augmented history: $\ell_{q,n,t,\ell}^{\text{skill}} = \log \pi_\theta(a_{q,n,t,\ell} \mid \tilde{h}_{q,n,t}, a_{q,n,t,<\ell})$ℓq,n,t,ℓskill​=logπθ​(aq,n,t,ℓ​∣h~q,n,t​,aq,n,t,<ℓ​) The ordinary student branch log-probability under the original interaction history is: $\ell_{q,n,t,\ell}^{\theta} = \log \pi_\theta(a_{q,n,t,\ell} \mid h_{q,n,t}, a_{q,n,t,<\ell})$ℓq,n,t,ℓθ​=logπθ​(aq,n,t,ℓ​∣hq,n,t​,aq,n,t,<ℓ​) The authors define the detached skill-induced log-probability shift $\Delta_{q,n,t,\ell} = \text{sg}[\ell_{q,n,t,\ell}^{\text{skill}} - \ell_{q,n,t,\ell}^{\theta}]$Δq,n,t,ℓ​=sg[ℓq,n,t,ℓskill​−ℓq,n,t,ℓθ​], where $\text{sg}[\cdot]$sg[⋅] denotes stop-gradient. This shift is mapped to a confidence gate $g_{q,n,t,\ell} = \sigma(\beta_{\text{opd}} \Delta_{q,n,t,\ell})$gq,n,t,ℓ​=σ(βopd​Δq,n,t,ℓ​). The OPD loss is defined as a confidence-gated sampled-token distillation objective: $\mathcal{L}_{\text{opd}}(\theta) = \mathbb{E}_{q,n,t,\ell} \left[ m_{q,n,t,\ell} \cdot g_{q,n,t,\ell} \cdot (\text{sg}[\ell_{q,n,t,\ell}^{\text{skill}}] - \ell_{q,n,t,\ell}^{\theta}) \right]$Lopd​(θ)=Eq,n,t,ℓ​[mq,n,t,ℓ​⋅gq,n,t,ℓ​⋅(sg[ℓq,n,t,ℓskill​]−ℓq,n,t,ℓθ​)] Because both the gate and the teacher log-probability are detached, minimizing this objective increases the ordinary policy's likelihood of teacher-endorsed on-policy tokens, internalizing the skill's behavioral effect.

In addition to OPD, SEED optimizes the policy with a group-relative RL objective. For each group $\mathcal{G}_q$Gq​, the trajectory-level group-relative advantage is computed as $A_{q,n}^{\text{rl}} = (R(\tau_q^{(n)}) - \mu_q) / (\sigma_q + \epsilon)$Aq,nrl​=(R(τq(n)​)−μq​)/(σq​+ϵ). The final training objective combines environment-driven agentic RL with hindsight-skill OPD: $\mathcal{L}_{\text{SEED}}(\theta) = \mathcal{L}_{\text{rl}}(\theta) + \lambda_{\text{opd}} \mathcal{L}_{\text{opd}}(\theta)$LSEED​(θ)=Lrl​(θ)+λopd​Lopd​(θ) The updated policy then becomes $\pi_{\theta_{\text{old}}}$πθold​​ for the next iteration, closing the self-evolving loop. At inference time, the deployed agent acts only from the ordinary interaction history $a_t \sim \pi_\theta(\cdot \mid h_t)$at​∼πθ​(⋅∣ht​). All hindsight skills are used only as training-time guidance, meaning deployment requires no analyzer, no skill bank, no retrieval module, and no augmented decision prompt.

Experiment

SEED is evaluated on three long-horizon agentic benchmarks—ALFWorld, WebShop, and Search-based QA—covering embodied control, web navigation, and tool-augmented question answering. Compared to outcome-only RL and static distillation baselines, SEED's dense token-level hindsight supervision from a self-evolving trajectory analyzer yields consistent improvements across all domains. Experiments show that internalizing skills through distillation is more effective than providing them as prompts, that the method accelerates convergence and reduces unproductive exploration, and that it achieves superior sample efficiency and generalization to unseen environments. Ablations confirm that each component—hindsight-skill SFT, self-evolving on-policy distillation, and on-policy skill extraction—is critical, with on-policy skills being the most important.

SEED substantially improves over outcome-only reinforcement learning by providing dense token-level hindsight supervision, with consistent gains across ALFWorld, Search-based QA, and WebShop. Internalizing skills into the policy yields better performance than supplying them as prompts during evaluation, and self-evolving distillation from the policy's own trajectories outperforms static distillation approaches. SEED outperforms GRPO and Skill-GRPO on all three long-horizon benchmarks, with especially large margins on ALFWorld and WebShop. Self-evolving hindsight distillation, which adapts to the policy's changing behavior, proves more effective than using a fixed skill library.

Ablation experiments on ALFWorld show that all three core components of SEED are essential for its strong average performance. The largest degradation comes from replacing on-policy skill distillation with a static offline library, which lowers the average by over 7 points. Removing hindsight-skill SFT or self-evolving OPD also causes substantial drops of around 5–6 points, confirming that initial trajectory analysis and continuous distillation from the evolving policy are both critical. Replacing on-policy skills with a static offline library causes the largest performance drop, reducing the average from 91.8 to 84.4. Removing hindsight-skill SFT decreases the average by 5.8 points, showing that initial trajectory-analysis capability provides an important foundation for self-evolution.

The evaluation uses three long-horizon benchmarks—ALFWorld, Search-based QA, and WebShop—to validate SEED’s dense token-level hindsight supervision and self-evolving skill distillation. Internalizing skills into the policy outperforms using them as prompts, and self-evolving distillation from the policy’s own trajectories surpasses static approaches. Ablation studies confirm that all three core components (on-policy skill distillation, hindsight-skill SFT, and self-evolving OPD) are essential, with the largest drop from replacing on-policy skills with a static library.