One-sentence Summary

Researchers from Nankai University, Northwestern Polytechnical University, and the Chinese Academy of Sciences introduce OraRL, a reinforcement learning post-training framework for video MLLMs that treats annotations as oracle rollouts and employs a decoupled advantage estimator to prevent advantage inversion, enabling efficient and scalable training without chain-of-thought while raising temporal mIoU from 62.5 to 66.0, tracking AO from 73.0 to 78.2, segmentation from 64.3 to 70.4, the three-benchmark spatial intelligence macro average from 51.0 to 56.1, and VSI-Bench score to 73.1 versus 55.0 for GPT-5 and 55.1 for Gemini-3-Pro.

Key Contributions

• The paper introduces OraRL, a reinforcement learning post-training framework for unified video MLLMs that serializes each annotation as an oracle rollout, turning it into a direct positive optimization target while preserving on-policy exploration.
• The framework identifies and resolves advantage inversion using a decoupled advantage estimator in which policy rollouts determine an oracle-free baseline and the oracle-policy gap calibrates a directional gain plus a separate detached oracle advantage.
• OraRL introduces sign-balanced pruning with moment correction, reducing step time from 92.5 to 62.4 seconds with an average loss of 0.4 points, and outperforms SFT and GRPO across controlled tasks while scaling from 0.8B to 9B models and up to 100k prompts. Video-ORA-9B decodes without chain-of-thought in 130 ms instead of 4,780 ms and improves temporal mIoU from 62.5 to 66.0, tracking AO from 73.0 to 78.2, segmentation from 64.3 to 70.4, and VSI-Bench score to 73.1.

Introduction

Unified video perception requires models to deliver precise temporal localization, spatial grounding, segmentation, tracking, and spatial understanding beyond coarse descriptions. Generalist multimodal large language models (MLLMs) are increasingly combining these capabilities, but they still lag behind task-specific experts in fine-grained perception, a gap that persists even with proprietary scale and additional post-training prompts. The central problem is sample efficiency: supervised fine-tuning treats annotations strictly as maximum-likelihood targets without task-level contrast, while group-based video RL methods use annotations only as scoring references and rarely obtain on-policy rollouts that match the precise annotations, leaving many optimization groups without a reliable positive anchor. The authors introduce OraRL, a reinforcement learning framework that serializes each human annotation as an oracle rollout, providing a dependable positive target. By excluding this oracle from the advantage baseline and coupling it with sign-balanced pruning, OraRL avoids the advantage inversion that cripples naive oracle mixing, enabling efficient, task-agnostic training that outperforms both SFT and standard GRPO across multiple model scales, data budgets, and seven video perception task families.

Method

The authors propose OraRL, a reinforcement learning framework that integrates ground-truth annotations directly into the policy optimization process. Given a multimodal query $q = (v, x)$q=(v,x), the policy model samples $n$n on-policy rollouts $\mathcal{O}_{op} = \{o_i\}_{i=1}^n$Oop​={oi​}i=1n​. Instead of discarding the human annotations, OraRL serializes each annotation $y$y into the model's response format to create an oracle rollout $o_{gt} = T_{task}(y)$ogt​=Ttask​(y). This oracle is appended to the on-policy group, forming an augmented set $\mathcal{O}_{aug} = \mathcal{O}_{op} \cup \{o_{gt}\}$Oaug​=Oop​∪{ogt​}.

As shown in the figure below:

Standard group relative policy optimization computes advantages by normalizing over the entire augmented group. However, incorporating the high-reward oracle into the baseline raises the advantage threshold, causing above-average on-policy rollouts to receive negative advantages and be penalized. To resolve this advantage inversion, OraRL computes the baseline exclusively from on-policy rewards and encodes the oracle-policy gap through specialized scaling terms.

First, the authors define the on-policy advantage estimation. The base advantage for each on-policy rollout is calculated as $A_i^{(0)} = r_i - \mu_{op}$Ai(0)​=ri​−μop​, where $\mu_{op}$μop​ is the mean of the on-policy rewards. By omitting variance normalization and excluding the oracle from the mean, any rollout outperforming the on-policy average is guaranteed a non-negative advantage. To capture the oracle-policy discrepancy, they compute a directional gain $g_q$gq​ based on the change in reward dispersion when the oracle is included:

This gain is applied exclusively to rollouts with positive base advantages ($A_i^{(0)} > 0$Ai(0)​>0), amplifying their signal without affecting below-average rollouts. The resulting advantages are then recentered to ensure a zero sum across the on-policy group, yielding $A_i^{op}$Aiop​.

Second, the framework introduces a detached oracle advantage. Rather than using the raw reward difference, the authors derive a calibrated scale $A_{gt}$Agt​ that adapts to the remaining supervision gap. They compute a normalized weight $w_q$wq​ that approaches zero as the on-policy mean converges to the oracle reward. This weight is combined with the maximum positive on-policy advantage $A_{max}^+$Amax+​ to bound the oracle's influence:

This ensures the oracle provides a strong anchor without dominating the gradient updates.

To improve computational efficiency, OraRL employs sign-balanced advantage pruning. Before the policy forward and backward passes, the method retains a subset of $K$K rollouts. The oracle is always kept as a positive anchor. The remaining slots are divided evenly between positive and negative on-policy rollouts, ranked by the absolute value of their sequence-level advantages. This sign quota preserves both reinforcing and suppressive signals, unlike magnitude-only pruning which might select only one sign.

Following pruning, post-selection moment correction is applied to restore statistical properties. Because the retained set favors large-magnitude rollouts and always includes the positive oracle, the mean and scale of the advantages shift. The authors first recenter the advantages to a zero sum. If centering would penalize the oracle, they project the vector to enforce a non-negative oracle advantage while maintaining the zero-sum constraint. Finally, the advantages are rescaled using a clipping factor $\lambda_q$λq​ that matches the root mean square of the selected subset to the pre-pruning on-policy RMS, preventing disproportionate update scales.

The unified policy update then utilizes these corrected advantages $\widehat{A}_i$Ai​. The objective function maximizes the likelihood of the oracle and positive policy rollouts while suppressing negative ones, using the standard clipped importance ratio $\rho_{i,t}$ρi,t​:

where $N_q$Nq​ is the total number of retained response tokens. This pipeline ensures that the policy learns from reliable positive targets while maintaining stable and balanced gradient updates.

Experiment

The experimental setup evaluates OraRL across seven video perception and spatial intelligence task families, comparing against SFT, GRPO, and other RL paradigms under controlled training budgets. OraRL achieves consistent improvements on all tasks, scaling with model size and data, while ablations show that its oracle-driven directional gain and detached advantage prevent the advantage inversion that plagues naive oracle injection. Additional analyses confirm that video data benefits spatial reasoning, chain-of-thought is unnecessary, and the method generalizes across backbones with efficient training and inference.

Video-ORA-9B achieves top temporal grounding results across all three TimeLens benchmarks, outperforming proprietary Gemini-2.5-Pro and the specialist TimeLens2-8B. The margin over TimeLens2-8B grows at stricter overlap thresholds, reaching a 6.7-point R1@0.7 gain on ActivityNet, which aligns with the benefit of annotation-supplied precise boundaries. A smaller 4B variant also ranks second among open models below 10B parameters, behind only TimeLens2-8B. Video-ORA-9B leads every metric on Charades-TimeLens, ActivityNet-TimeLens, and QVHighlights-TimeLens, improving mIoU over TimeLens2-8B by 2.3 to 5.0 points and exceeding Gemini-2.5-Pro. On ActivityNet and QVHighlights, the gap over TimeLens2-8B widens as the required temporal overlap tightens, with the largest R1@0.7 advantage of 6.7 points on ActivityNet. Video-ORA-4B ranks second in mIoU among open models under 10B parameters, behind only TimeLens2-8B.

Video-ORA-9B leads spatial grounding on all eight RefCOCO, RefCOCO+, and RefCOCOg validation and test splits, outperforming the strongest baseline by 0.5 to 2.7 R@0.5 points. The largest margin appears on RefCOCO+ validation, where absolute location words are excluded. This pattern is consistent with stronger grounding from visual appearance and object relations. Video-ORA-9B ranks first in R@0.5 on every split in the RefCOCO family. The margin over the strongest baseline is widest on RefCOCO+ validation, where absolute location words are absent.

Video-ORA-9B achieves the best results among open-source models on five of seven multiple-choice video question answering benchmarks, substantially improving over its Qwen3.5-9B backbone. The largest gains are on VideoHolmes, VideoMME, and MMVU, while LongVideoBench sees only a modest increase. Video-ORA-9B ranks first on five of the seven evaluated benchmarks. The macro average across benchmarks rises from 61.9 to 66.8 compared to the backbone. Performance jumps are most pronounced on VideoHolmes, VideoMME, and MMVU. LongVideoBench shows a minimal improvement of 1.6 points.

On GOT-10k tracking with 32 frames, Video-ORA-9B achieves the best average overlap and recall at every IoU threshold among all reported models. Video-ORA-4B also outperforms the Qwen3.5 models of comparable scale but remains below OneThinker-8B and Video-ORA-9B. The lead of Video-ORA-9B over OneThinker-8B is largest at the strictest localization threshold. Video-ORA-9B leads all listed models on average overlap and all recall thresholds. Both Video-ORA models show substantial tracking gains over their Qwen3.5 counterparts. The largest gap between Video-ORA-9B and the next best model occurs at the strictest IoU threshold.

On spatial-temporal video grounding, Video-ORA-9B leads all compared models in temporal and spatial localization. Its spatial grounding advantage over the Qwen3.5-9B backbone is especially large. Video-ORA-4B also improves over Qwen3.5-4B and surpasses the larger Qwen3.5-9B on temporal metrics while achieving stronger spatial scores. Video-ORA-9B achieves the highest temporal and spatial localization scores among the compared models. Video-ORA models show larger relative gains in spatial grounding than in temporal grounding, especially over their Qwen3.5 backbones.

Video-ORA is evaluated across temporal grounding, spatial grounding, video question answering, object tracking, and spatio-temporal grounding, where the 9B model consistently achieves state-of-the-art results, outperforming proprietary models like Gemini-2.5-Pro and specialist baselines. The model benefits from precise boundary annotations, with gains widening at stricter overlap thresholds, and shows larger relative improvements in spatial grounding over its Qwen3.5 backbone. A smaller 4B variant also demonstrates strong performance, ranking second among open models under 10B parameters in several tasks.