One-sentence Summary

Researchers from Shanghai Jiao Tong University, Shanghai AI Laboratory, and collaborators introduce PAWBench and PAWEval, a benchmark and protocol that assess video generators as probabilistically aligned world models by evaluating outcome distributions rather than single-video plausibility, and across 50 scenarios no current model consistently matches reference probabilities, revealing a critical gap.

Key Contributions

• Probabilistic alignment is formalized as a distributional criterion requiring that repeated video generations under the same initial observation and action reproduce the correct distribution over possible physical futures.
• PAWBench, a benchmark of 50 controlled stochastic scenarios, is introduced, and evaluation of eleven video generators shows that no existing model consistently matches reference probabilities or recovers the full range of valid behaviors.
• PAWEval, an outcome-level protocol, is developed to map repeated video rollouts to empirical outcome distributions, enabling tractable distributional comparison across calibration and coverage regimes.

Introduction

Recent advances in video generation have positioned these models as potential world models that can simulate plausible futures from an initial observation and an action. However, a true world model must capture the full distribution of possible outcomes, not just a single plausible continuation, a requirement the authors term probabilistic alignment. Existing benchmarks evaluate individual videos on quality and coherence but do not test whether the model’s generated distribution matches the physically valid set of futures. The authors introduce PAWBench, a benchmark of 50 stochastic physical scenarios designed to measure how well video generators recover the correct outcome probabilities and coverage of valid futures. They find that current models produce plausible individual rollouts yet fail to align their output distributions with the underlying physical processes, revealing a critical gap in world modeling capabilities.

Dataset

The authors introduce PAWBench, a benchmark for evaluating probabilistic alignment in video generation models. It contains 50 manually curated scenarios, each providing a source image (the initial observation), an action prompt, a set of valid terminal outcomes, and, for calibration scenarios, a reference distribution. Scenarios span eight mechanism groups: tossing, rotation, routing, draw-style randomizers, collision, stability, agent interaction, and material transition.

The benchmark is split into two complementary subsets:

• PAW-Calibration (25 scenarios): Reference distributions are derived analytically or from physical symmetry. It measures whether the model’s empirical outcome distribution approaches the reference, detecting probability misallocation.
• PAW-Coverage (25 scenarios): Outcomes are enumerable but their probabilities are not reliably specified. It checks whether repeated rollouts recover all distinct valid futures, detecting missing outcomes.

Each scenario is built under three requirements: stochasticity must arise from a visible physical mechanism, the action prompt must describe one atomic intervention, and terminal outcomes must form a finite, visually distinguishable set. The authors manually generate the source image and action prompt, list the valid outcomes, define failure criteria, and, for PAW-Calibration, derive the reference distribution. A rubric-based judging protocol called PAWEval maps generated videos to terminal outcomes.

For evaluation, the model is queried K times with the same source image and action prompt. The resulting videos are mapped to outcomes, producing an empirical outcome distribution. In PAW-Calibration, this distribution is compared to the reference; in PAW-Coverage, coverage of the outcome set is assessed. The design separates calibration and coverage failures that single-sample evaluation cannot distinguish.

Method

The authors propose a framework for probabilistically aligned world modeling, which treats a world model as a conditional distribution $P_M(\tau \mid x, a)$PM​(τ∣x,a) over future trajectories $\tau$τ given an initial observation $x$x and action $a$a. This formulation captures the stochastic nature of physical processes, where identical initial conditions can lead to multiple valid outcomes. The authors define two levels of alignment: support alignment, which requires the model to realize all distinct possible outcomes, and probability-mass alignment, which demands that these outcomes occur in their correct relative proportions.

To achieve probabilistic alignment, the authors intervene at three interfaces that shape the distribution of generated futures: language prompting, initial noise sampling, and model parameter updates.

First, the authors explore prompt engineering to steer the video generator via language. They evaluate three settings: direct vision-language model sampling, where models act as samplers over possible futures; standard prompt engineering, where a language model predicts an outcome and writes a corresponding generator prompt; and Oracle prompt engineering, where target outcomes are explicitly specified in the prompts to follow a reference distribution. This approach isolates errors arising from the controller selecting the wrong distribution versus the generator failing to produce the requested outcome. Even when manually specified targets are provided through Oracle prompt engineering, the generators struggle to realize the requested outcomes.

As shown in the figure below:

The results indicate that current language-based control is limited, as generators realize only a fraction of the supplied targets, highlighting the difficulty of reliably aligning the output distribution through prompting alone.

Second, the authors investigate initial noise sampling to determine if the observed distributional gaps stem from finite-sample exploration failures. Holding the action prompt and generator fixed, they employ Couple to Control, a repulsive Gaussian coupling scheme. This method introduces negative dependence among the initial noise samples while preserving the standard Gaussian marginal for each sample. By doing so, the authors test whether independent draws repeatedly visit the same modes and whether coupling the noise can encourage the exploration of other valid outcomes within a fixed rollout budget.

Third, the authors examine updating model parameters to alter the learned allocation of probability mass. They train LoRA-adapted video generation models on datasets with varying ratios of specific outcomes. By adjusting the training data composition while keeping the training budget and recipe fixed, they probe how global outcome frequencies influence the model's conditional distribution. The authors find that changing the global mixture provides only coarse control, as adjusting the weights for one scene often degrades the alignment for another. This demonstrates that true probabilistic alignment requires the model to learn how the distribution of possible futures should dynamically change based on the initial physical state of each specific scene.

Experiment

PAWBench evaluates video generation models as probabilistic world models by repeatedly rolling out the same initial observation and action, then measuring the resulting outcome distribution across two tracks: PAW-Calibration tests whether the empirical distribution matches a known reference, while PAW-Coverage checks whether all valid outcomes appear. The experiments reveal that current models fail both to allocate probability correctly and to recover the full set of plausible futures, and they do not reliably distinguish physically causal interventions from non-causal input changes. Further analysis shows that neither increasing the number of rollouts nor using language-based prompt engineering resolves these shortcomings, as models struggle both to select the right distribution of outcomes and to faithfully realize requested futures in video.

Current video generators show a clear separation between probabilistic calibration and support coverage: the model with the lowest calibration error passes only a fraction of scenes, and the model with the highest coverage average similarly leaves many scenes behind. Even models that pass all scenes on one metric do not lead that metric, underscoring that scene-level reliability and conditional alignment capture distinct failure modes. The best-calibrated model (Cosmos 3 Super I2V) passes only 80% of calibration scenes, while the highest-coverage model (LTX-2.3) passes only 72% of coverage scenes. Seedance 2 passes every calibration scene but does not achieve the lowest calibration error, and LingBot-Video-MoE passes every coverage scene but does not reach the highest coverage. Models that recover broad support (high Coverage) are not the ones that best match reference probabilities (low Calibration TVD), revealing that these two facets of world modeling are decoupled. Scene pass rates vary widely, with some models failing to produce readable outcomes for a substantial share of scenes, limiting the reliability of their conditional averages.

Direct sampling of future outcomes from vision-language models yields distributions that are misaligned with reference probabilities. Among the five evaluated VLMs, GLM-5V Turbo achieves the lowest calibration error while Gemini 3.5 Flash attains the highest coverage, but no model performs well on both metrics simultaneously. GLM-5V Turbo produces the most calibrated outcome distribution, with the lowest total variation distance from the reference. Gemini 3.5 Flash covers the widest range of possible outcomes, achieving the highest coverage percentage. All models exhibit a trade-off between calibration and coverage, with none excelling at both.

GPT-5.5's prompt-engineered outcome selections are misaligned with the target distribution before any video is generated. When these selections are passed to video generators, calibration error increases for all models, while coverage improves for only some. Manually specified target outcomes (Oracle PE) consistently reduce calibration error and improve coverage, yet generators still fail to realize a large share of the requested outcomes, revealing shortcomings in both the controller and the generation process. GPT-5.5 PE alone selects outcomes with a calibration TVD of 44.3 and coverage of 35.0%, already far from the reference distribution. Adding PE to video generators raises calibration TVD in every case (e.g., Cosmos 3 Super I2V from 20.5 to 31.6) and improves coverage for only some models, while Wan2.2 coverage drops from 63.4% to 61.9%. Oracle PE lowers calibration TVD and raises coverage for every generator, but even then generators realize only 37.6–58.1% of the requested outcomes.

Coupling the noise across 50 rollouts with C2C reduces mean calibration error and increases mean coverage for all three tested video generators, indicating broader exploration of each model's existing output distribution. The improvements vary by mechanism and do not consistently raise the success probability ratio, suggesting that C2C diversifies the generated futures without altering the underlying learned distribution. C2C lowers mean Calibration TVD for Wan2.2, LTX-2.3, and Cosmos 3 Super I2V, with LTX-2.3 showing the largest reduction (from 30.1 to 19.9). Mean Coverage improves under C2C across all generators, with Cosmos 3 Super I2V gaining the most (from 55.2% to 63.9%). SPR changes are inconsistent: Wan2.2's calibration SPR rises from 64% to 84%, while its coverage SPR drops from 92% to 88%.

Training a video generator on mixtures with increasing proportions of left-falling pencil examples shifts the outcome distribution for both upright and left-leaning scenes in the same direction. A model trained with 20% left-fall videos best approximates the 50/50 reference for an upright pencil but leaves the left-leaning scene at chance, while training with 80% or more left-fall videos matches the 100/0 reference for the left-leaning scene but causes the upright pencil to fall left almost always. No single training mixture can simultaneously satisfy both reference distributions. Increasing the share of left-fall training videos shifts outcome mass leftward for both upright and left-leaning scenes, moving their distributions in lockstep. Training on 80% or 100% left-fall videos perfectly matches the left-leaning reference (TVD 0.0), but the upright scene then falls left almost every time (TVD 48.0–50.0).

The evaluation assesses how well video generators and vision-language models produce outcome distributions aligned with reference probabilities across diverse scenes, measuring both calibration error and coverage. Across all models, calibration and coverage are decoupled: no single model excels at both, and prompt engineering via a language model consistently worsens calibration while only occasionally improving coverage. Coupling noise across rollouts improves both metrics but does not alter the underlying learned distribution, while training on skewed data mixtures shifts outcome distributions uniformly, preventing any single model from satisfying multiple reference distributions simultaneously.