One-sentence Summary

Researchers from Tianjin University, Shanxi University, and Imperial College London propose WarpSAC, a regime-aware off-policy RL family that adapts stabilizers to data availability via Sample Weight Decay and the prescribed variants WarpSAC-L and WarpSAC-A, improving normalized score–step AUC over FlashSAC by $4.5\%$4.5% across nine CPU-scale environments and $23.1\%$23.1% across fourteen GPU-parallel environments, lifting UnitreeG1TransportBox-v1 success from $19.8\%$19.8% to $96.4\%$96.4%, and accelerating sim-to-real deployment on Unitree G1 by $36.4\%$36.4% in wall time.

Key Contributions

• Formalizes a data-regime hypothesis for off-policy RL, showing that stabilizer utility (parameter normalization, clipped double-Q) depends on replay coverage, validated via controlled ablations across eight benchmark families spanning CPU-scale locomotion, GPU-parallel robotic simulation, dexterous manipulation, and humanoid whole-body control.
• Introduces WarpSAC, a regime-aware off-policy RL family with two variants: WarpSAC-L (norm ON, clipped double-Q) for data-limited CPU-scale training and WarpSAC-A (norm OFF, single-Q) for data-abundant GPU-parallel training, with Sample Weight Decay as the regime-agnostic core component.
• WarpSAC improves normalized score-step AUC over FlashSAC by 4.5% across nine CPU-scale environments and 23.1% across fourteen GPU-parallel environments, lifts UnitreeG1TransportBox-v1 success rate from 19.8% to 96.4%, gains 19.1% in mean normalized wall-time AUC on MuJoCo Playground, and achieves 36.4% faster sim-to-real deployment on Unitree G1 in wall time.

Introduction

Modern reinforcement learning for robot control increasingly relies on GPU-accelerated simulators and massively parallel environments, which generate far more interaction data than traditional single-environment setups. This abundance makes off-policy actor-critic methods like Soft Actor-Critic attractive, yet most existing stabilizers, such as entropy regularization, clipped double-Q targets, and parameter normalization, were designed for narrow replay coverage where exploration and conservative value estimation are critical. In data-rich regimes, these components can restrict expressive freedom, introduce pessimism, and add computational overhead, but scalable pipelines still inherit them without question.

The authors investigate when these stabilizers help versus hinder by conducting a controlled component-wise analysis using FlashSAC. They isolate three axes: replay-side data utilization via Sample Weight Decay, parameter-projection normalization, and critic multiplicity between clipped double-Q and single-Q targets. Across eight benchmark families and 67 environments, they find that normalization aids narrow replay but limits value fitting with broad replay, clipped double-Q can be relaxed in high-throughput manipulation, and SWD remains consistently useful. Based on this, they build WarpSAC, a regime-aware algorithm family that matches stabilizer choices to data availability, achieving significant improvements in score-step AUC, success rates, and wall-clock time over FlashSAC without altering the training backbone.

Method

Method

Preliminaries

The authors build on the standard discounted Markov decision process (MDP) formulation for continuous-control reinforcement learning, where the goal is to maximize the expected discounted return. Their study extends Soft Actor-Critic (SAC), a maximum-entropy off-policy actor-critic algorithm. SAC augments the return objective with an entropy bonus, and maintains a replay buffer, a stochastic policy, and two critics to counter overestimation via the clipped double-Q target. The clipped double-Q operator suppresses spuriously high Q estimates on poorly covered actions, but introduces a second critic and a pessimism bias; a single-Q variant removes the minimization and uses only one critic.

FlashSAC extends SAC to large-scale robotic control by combining high-throughput data collection, larger models, reduced update frequency, and norm-control mechanisms for stable critic learning. Parameter projection normalization, a key component, constrains each layer's weight matrix to a Frobenius-norm ball after each optimizer step. This bounds the spectral norm and, for networks with 1-Lipschitz activations, yields a Lipschitz upper bound on the entire network. This normalization acts as an exploration- and stability-oriented constraint: it controls the effective function class but may restrict expressive freedom when replay coverage is already sufficient.

The authors also formalize weighted replay, where each transition is sampled with a probability proportional to a sample weight. Different weight choices recover uniform sampling and prioritized replay. Sample Weight Decay (SWD) instantiates this idea with age-aware weights and was originally proposed as a lightweight replay-side method for mitigating plasticity loss. The authors study SWD both as a plasticity-preserving method and as a mechanism that facilitates data exploitation during off-policy learning.

Data-Regime Hypothesis

The authors argue that classical off-policy stabilizers were developed under CPU-scale assumptions, where replay coverage is narrow and value extrapolation is fragile. GPU-parallel simulation changes this data regime: thousands of actors populate the buffer with diverse trajectories at high throughput, shifting the bottleneck from obtaining sufficient coverage to fitting and exploiting high-value behavior from abundant data. The authors hypothesize that this shift changes the relative value of the classical stabilizers. In the data-limited regime, parameter normalization and clipped double-Q help by constraining the effective function class and suppressing spuriously high Q values on poorly covered actions, while replay-side mechanisms should make better use of the narrow buffer. In the data-abundant regime, the same conservative mechanisms can restrict value fitting or add unnecessary pessimism, while replay-side gains persist because targeting policy-relevant transitions is orthogonal to coverage.

The Three Axes

Given this hypothesis, the authors isolate three design axes, all on top of FlashSAC:

• Replay weighting $w_t(i)$wt​(i): whether transitions are sampled uniformly or with an age-dependent weight.
• Parameter projection normalization: whether the FlashSAC column-wise weight renormalization is applied after each optimizer step (Norm ON) or disabled (Norm OFF).
• Critic multiplicity: whether the clipped double-Q target is used (two critics) or replaced by a single critic (Single-Q).

The authors fix data-collection throughput and optimizer schedule to compare regimes cleanly, and vary network capacity only in a separate scale ablation. Varying these three axes independently enables regime-aware pairing rather than collapsing WarpSAC into a single "strong" or "weak" SAC recipe.

Sample Weight Decay

Among the three axes, replay weighting is the only one applied regardless of data regime. SWD instantiates this axis by biasing minibatch sampling toward recent transitions using a linear age decay. For a transition inserted at time $t_i$ti​, its age at training step $t$t is $A_t(i) = t - t_i$At​(i)=t−ti​. SWD assigns each transition the age-dependent weight

where $T_{\text{decay}}$Tdecay​ is the decay horizon and $w_{\min} > 0$wmin​>0 is a floor that prevents old transitions from being fully discarded. Setting $T_{\text{decay}} = 0$Tdecay​=0 recovers uniform replay. SWD is implemented inside the replay buffer with no auxiliary networks, additional Bellman targets, or loss changes.

The authors note that policy performance is dominated by Bellman errors on state-action regions visited by the current policy and on regions along high-value trajectories. Uniform replay ignores this structure and spends equal update probability on transitions from much older policies. SWD redirects a fixed update budget toward more policy-relevant transitions by changing only the minibatch distribution, without touching the nominal update-to-data ratio or the update rule; the nonzero floor $w_{\min}$wmin​ preserves coverage. Because this argument depends on policy age rather than data volume, SWD is retained in both regimes.

Regime-Aware Variants and Prescription

With SWD established as the regime-agnostic replay component, the remaining design question is how to set normalization and critic multiplicity for each data regime. The authors define two prescribed variants: WarpSAC-L for the data-limited (CPU-scale) regime, which keeps normalization and conservatism to stabilize value extrapolation under narrow replay coverage, and WarpSAC-A for the data-abundant (GPU-parallel) regime, which removes normalization and conservatism to free the critic to exploit broad, rapidly refreshed replay data. An intermediate ablation point, WarpSAC w Norm OFF, is also defined. FlashSAC (no SWD, Norm ON, clipped double-Q) serves as the shared baseline.

These variants form a conservatism spectrum, from full (Norm ON, double-Q) to minimal (Norm OFF, single-Q), with SWD applied throughout. A key feature of WarpSAC is that the GPU-parallel recipe achieves gains by removing components rather than adding them: dropping normalization frees the critic to fit abundant data, and dropping the second critic halves critic-side computation. The result is an algorithm that is both simpler and stronger than the fully stabilized baseline. This stands in contrast to the common pattern of stacking mechanisms for robustness; WarpSAC shows that matching stabilizers to the data regime is more effective than uniform inheritance. The practitioner's guide is straightforward: use WarpSAC-L for data-limited regimes, WarpSAC-A for data-abundant regimes, and always enable SWD.

Experiment

The experiments validate the data-regime hypothesis for the WarpSAC family across eight benchmark families in both CPU-scale and GPU-parallel settings. In data-limited environments, WarpSAC-L, which pairs SWD with normalization, excels by stabilizing value learning under scarce replay, while in data-abundant settings, WarpSAC-A shows that conservative stabilizers like normalization and clipped double-Q become restrictive and can be relaxed. A sim-to-real case study on Unitree G1 demonstrates WarpSAC achieves faster deployment than FlashSAC, and mechanism analyses reveal that SWD compensates for limited network capacity in CPU-scale training, while normalization constrains expressivity in GPU-parallel settings, with the best results coming from combining SWD with reduced normalization.

WarpSAC adapts its normalization and critic settings based on data regime, with SWD replay applied across all variants. In data-limited settings, normalization and double critics are retained, while in data-abundant settings, normalization is disabled and a single critic is used, reducing computation and improving performance. This regime-aware design consistently outperforms uniform stabilizer inheritance. SWD provides the largest gains in low-capacity settings, with relative improvements exceeding 2x on some tasks. Disabling normalization sharply improves performance in GPU-parallel settings with limited capacity, and remains competitive as capacity grows. Pairing SWD with reduced normalization yields the strongest configurations in data-abundant environments. WarpSAC achieves gains by removing components rather than adding them, simplifying the algorithm while improving strength.

WarpSAC uses a data regime aware configuration, keeping normalization and double critics in low data settings but disabling normalization and using a single critic when data is abundant, which cuts computation and boosts performance. This adaptive approach consistently beats uniform stabilizer inheritance, with SWD replay delivering the largest gains in low capacity tasks, sometimes more than doubling relative performance. In GPU parallel settings with limited capacity, removing normalization sharply improves results and stays competitive as capacity grows, while combining SWD with reduced normalization yields the best outcomes in data rich environments. Overall, WarpSAC improves by simplifying the algorithm rather than adding components.