One-sentence Summary

Researchers from Nanjing University and JD.com propose Cross-Teacher Guarding for multi-teacher on-policy distillation (ROAD MOPD), which introduces nonrouted teachers as auxiliary cross-domain critics and coordinates multi-teacher supervision through token-level credit assignment and preventive multi-teacher gradient projection, mitigating disadvantaged-domain degradation, gaining over 5 percentage points under high conflict, and improving low-conflict and three-domain performance.

Key Contributions

• The paper proposes ROAD MOPD, which uses non-routed teachers as auxiliary cross-domain critics on the same student trajectories and coordinates multi-teacher supervision through token-level credit assignment and gradient updates.
• The method introduces cross-teacher-guided credit assignment to strengthen token-level corrections jointly supported by teachers and attenuate conflicting signals, plus preventive multi-teacher gradient projection to constrain conflicting updates while adaptively retaining target-domain learning signals.
• Experiments show that ROAD MOPD mitigates degradation in disadvantaged domains, gains over 5 percentage points under high conflict, and improves overall performance in low-conflict and three-domain settings compared with standard MOPD and gradient-conflict correction baselines.

Introduction

Reinforcement learning for post-training large language models often depends on outcome-level rewards, which give little direct feedback on intermediate reasoning steps and make credit assignment difficult. On-policy distillation addresses this by providing dense token-level supervision from a strong teacher on student-generated trajectories, and multi-teacher on-policy distillation extends the idea by routing each domain sample to its specialized teacher. However, domain routing alone does not coordinate how updates from different domains affect shared parameters, and in high-conflict combinations such as medicine with math or code it can produce widespread gradient conflicts and asymmetric performance degradation. The authors propose ROAD MOPD, which uses non-routed teachers as cross-domain critics on the same student trajectories. It combines token-level weighting based on teacher agreement with preventive gradient projection, suppressing conflicting updates while preserving the routed teacher’s domain-specific learning signal.

Method

On-Policy Distillation with Multiple Domain Teachers

The authors formulate the learning problem around a student policy $\pi_\theta$πθ​, a set of teacher policies $\pi_T$πT​, and a prompt distribution $\mathcal{D}$D. For a prompt $x\sim\mathcal{D}$x∼D, the student samples a response $y=(y_1,\dots,y_T)$y=(y1​,…,yT​) and maintains prefixes $h_t=(x,y_{<t})$ht​=(x,y<t​). Standard on-policy distillation minimizes the sequence-level reverse-KL objective

Following prior work, this objective is expressed as relative-reward maximization with a KL penalty to a frozen reference policy $\pi_{\mathrm{ref}}$πref​. With reward scale $\lambda>0$λ>0, the maximization target is

where $\mathbb{E}_{\theta}$Eθ​ averages over $x\sim\mathcal{D}$x∼D and $y\sim\pi_\theta(\cdot\mid x)$y∼πθ​(⋅∣x). Setting $\lambda=1$λ=1 recovers standard on-policy distillation, while $0<\lambda<1$0<λ<1 attenuates the teacher signal and $\lambda>1$λ>1 amplifies it. The frozen reference $\pi_{\mathrm{ref}}$πref​ provides a shared baseline rather than an additional teacher.

For $K$K domain-teacher pairs $\{(\mathcal{D}_d,\pi_{T_d})\}_{d=1}^{K}${(Dd​,πTd​​)}d=1K​, the multi-domain on-policy distillation objective routes each prompt to its corresponding domain teacher and minimizes

For a fixed target domain $d$d, the routed teacher is denoted $T_{\mathrm{r}}=T_d$Tr​=Td​, and a teacher from an auxiliary domain is denoted $T_{\mathrm{a}}$Ta​. Both teachers and the frozen reference score the same student-generated tokens under teacher forcing, without additional rollouts. Their token-level log-probabilities are written as $\ell_t^{\mathrm{r}}$ℓtr​, $\ell_t^{\mathrm{a}}$ℓta​, and $\ell_t^{\mathrm{ref}}$ℓtref​.

Cross-Teacher-Guided Credit Assignment

A fixed global $\lambda$λ applies the same reward scale to every token, even though a trajectory may contain transferable reasoning patterns and domain-specific corrections. The routed teacher alone does not provide a cross-domain check on those corrections. To address this, the authors query an auxiliary teacher on the same student-generated prefixes and use its feedback to adjust the routed teacher's token-level reward. The auxiliary teacher is not used as a supervisor for solving the target-domain task; its relative preference acts only as a compatibility signal.

Token-level agreement between the routed teacher and the auxiliary teacher is measured relative to the shared frozen reference:

Positive $s_t$st​ means both teachers favor the same correction direction relative to the reference. Negative $s_t$st​ indicates opposing preferences, with values near $-1$−1 corresponding to strong cross-teacher disagreement. The score approaches zero when either teacher provides little evidence beyond the reference.

This agreement replaces the global reward scale with a token-wise coefficient:

where $\rho_{\mathrm{g}}>0$ρg​>0 controls modulation strength. When $s_t>0$st​>0, the routed teacher's relative reward is strengthened on tokens supported by both teachers. When $s_t<0$st​<0, the reward is attenuated, with stronger disagreement producing greater attenuation. When $s_t=0$st​=0, the method reduces to standard on-policy distillation, and clipping limits both amplification and attenuation.

For a sampled token $y_t$yt​ and a comparison policy $q$q, the $k_1$k1​ log-ratio estimator is

The modulated signal

is clipped and detached as the advantage in the PPO-style surrogate token loss $\ell_{d,i,t}^{\mathrm{agi}}(\theta)$ℓd,i,tagi​(θ). Gradients flow through the policy importance ratio rather than through the advantage. When multiple auxiliary teachers are used, their token-wise agreement scores with the routed teacher are averaged before reward modulation.

Preventive Multi-Teacher Gradient Projection

Token-level agreement does not guarantee compatible parameter updates, because token supervision signals are aggregated through shared parameters. The authors therefore formulate gradient correction as a constrained optimization problem: minimize conflict with an auxiliary teacher on the same trajectories while preserving a minimum fraction of the target-gradient norm.

First, the target-domain gradient is computed from the agreement-modulated loss:

To construct the auxiliary gradient, the routed teacher is replaced with the auxiliary teacher and agreement modulation is disabled. Let $\ell_{d\to\mathrm{a},i,t}^{\mathrm{OPD}}$ℓd→a,i,tOPD​ denote the resulting plain token-level on-policy distillation loss. The auxiliary loss uses the same response-level reduction as the target loss:

Both losses use the same stored domain-$d$d trajectories. Thus $g_d'$gd′​ reflects the auxiliary teacher's preferences on the current behavior, rather than its training gradient on auxiliary-domain data.

The corrected gradient is optimized separately for each trainable parameter tensor. Let $g$g and $g'$g′ denote nonzero slices of $g_d$gd​ and $g_d'$gd′​, and let $v$v denote a candidate corrected gradient. For an intended tensor-level step $-\eta v$−ηv, the first-order change in the auxiliary loss is $-\eta\langle v,g'\rangle$−η⟨v,g′⟩. A negative inner product $\langle g,g'\rangle<0$⟨g,g′⟩<0 indicates local conflict; otherwise, $g$g is retained. For conflicting pairs, the candidate is restricted to the line segment between $g$g and its full projection onto the hyperplane orthogonal to $g'$g′:

This restriction preserves the component orthogonal to $g'$g′ and removes only the conflicting component, up to full projection. Full projection eliminates local conflict but can nearly erase $g$g when the two gradients strongly oppose each other. The corrected gradient is therefore chosen to minimize residual conflict under a norm-retention floor $m\in(0,1]$m∈(0,1]:

The resulting solution can be written as a partial projection:

Here $\alpha^\star$α⋆ is the fraction of the conflicting component removed. Full projection with $\alpha^\star=1$α⋆=1 is used whenever it satisfies the norm floor; otherwise, as much conflict is removed as the floor permits. For non-conflicting pairs, $\alpha^\star=0$α⋆=0. These are local statements about the same-trajectory auxiliary loss, not guarantees of auxiliary-domain performance after an optimizer step. Tensors with negligible gradient norm are left unchanged, and auxiliary gradients are never applied as separate optimizer updates. With multiple auxiliary teachers, the projections are applied sequentially under a shared norm-retention floor relative to the original target gradient.

The authors contrast this preventive projection with post-hoc gradient surgery methods such as PCGrad. Here, the auxiliary teacher assesses the current target-domain trajectory and constructs a gradient constraint specifically for that update, constraining potentially incompatible corrections while retaining target-domain learning signals.

Overall Objective and Training

The framework uses sequence-mean aggregation: token losses are averaged within each response, then across responses, and finally across domains. Auxiliary losses use the same reduction but guide only the gradient correction, so the overall distillation objective includes only target-domain losses:

After the preventive gradient projection, only the projected target-domain gradients are averaged, using the same domain weights as in the overall objective. The resulting update direction

is globally clipped and used for one AdamW optimizer step.

Experiment

The evaluation covers Code, Math, and Medical benchmarks using Qwen3-4B and Qwen3-1.7B students trained from multiple domain teachers, comparing ROAD MOPD with several multi-domain policy distillation baselines. ROAD MOPD consistently achieves the best two-domain averages on Math-Medical and Code-Medical pairs, recovering Medical accuracy while preserving or improving gains in the other domain, and it also generalizes to lower-conflict and three-domain settings. Further analyses show that cross-teacher disagreement tracks gradient conflict, preventive projection outperforms post-hoc PCGrad correction, and both credit assignment and gradient protection components contribute to joint performance.

ROAD MOPD achieves the highest two-domain average among multi-domain methods across the paired-domain settings for both 4B and 1.7B students, outperforming the strongest baseline in every case. With 4B students, it restores medical accuracy to near or above the single-domain reference while also improving the paired math or code domain. With 1.7B students, it surpasses the single-domain medical reference and its larger medical gains outweigh slightly lower math or code scores, improving the overall trade-off. ROAD MOPD attains the best overall two-domain average in all paired-domain and student-size settings, with consistent margins over the strongest multi-domain baseline. Vanilla MOPD degrades medical accuracy below the initial student, while ROAD MOPD recovers medical performance near or above the single-domain reference and leads the paired math or code domain in the larger-student setting. In the smaller-student setting, ROAD MOPD exceeds the single-domain medical reference and offsets smaller math or code deficits with substantially stronger medical gains.

ROAD MOPD improves joint performance on Qwen3-4B in the low-conflict Math × Code setting. It achieves the best Code average and second-best Math average among compared multidomain methods, and its larger Code gain offsets a smaller Math gap to produce the highest reported overall average. ROAD MOPD attains the best Code average and second-best Math average among the compared multidomain methods. The method's Code advantage is larger than its Math deficit relative to ExOPD, so it achieves the highest reported overall average.

In a three-domain evaluation spanning math, code, and medical tasks, the proposed method achieves the highest average accuracy in every domain and the highest overall mean. It improves over the single-teacher OPD baseline across all three domains and over the strongest multi-domain baselines overall. The results indicate balanced gains rather than trading off one domain for another. The proposed method leads all compared methods in math, code, and medical domain averages. Its overall mean exceeds the next best method by more than one point and improves over the single-teacher OPD baseline across every domain.

Ablations show that removing credit assignment or gradient protection reduces domain-average accuracy in both two-domain settings, with the largest declines in Math and Code for credit assignment and in Medical for gradient protection. Replacing adaptive retention with full projection also lowers the two-domain mean, despite a small domain-level gain in one setting. The complete ROAD MOPD method achieves the highest equally weighted two-domain average in both domain pairs. Credit assignment removal lowers accuracy across domains, with especially clear declines in Math and Code. Gradient protection removal reduces Medical accuracy, particularly in the Code-Medical setting. Adaptive retention yields better two-domain averages than full projection, even when full projection improves Medical in one pairing. The full method records the best two-domain average in both evaluated domain pairs.

Across paired-domain and three-domain evaluations with 4B and 1.7B student models, ROAD MOPD consistently achieves the best overall domain averages, restoring or surpassing single-domain medical performance while improving math and code. In the low-conflict Math and Code setting it leads overall through stronger code gains, and in the three-domain setting it attains balanced gains across all domains. Ablations confirm that credit assignment, gradient protection, and adaptive retention each contribute to the method's two-domain performance.