One-sentence Summary

FAIR at Meta, New York University, and Berkeley AI Research introduce Navigation World Model (NWM), a 1-billion-parameter Conditional Diffusion Transformer trained on diverse egocentric videos that plans trajectories by simulating and ranking them in familiar environments and leverages learned visual priors to imagine paths in unfamiliar environments from a single image, surpassing the fixed behavior of supervised navigation policies.

Key Contributions

• This paper introduces Navigation World Model (NWM), a controllable video generation model that predicts future egocentric observations from past observations and navigation actions using a Conditional Diffusion Transformer (CDiT) trained on large-scale human and robot egocentric videos and scaled to 1 billion parameters.
• This paper shows that NWM supports model-based navigation planning by simulating candidate trajectories and evaluating goal achievement, enabling both planning from scratch and ranking trajectories sampled from external policies with improved goal-conditioned visual navigation performance.
• This paper demonstrates that training on action- and reward-free egocentric video data such as Ego4D improves video prediction and generation in unseen environments, allowing NWM to imagine plausible navigation trajectories from a single input image.

Introduction

Visual navigation is a core capability for robotic agents because it requires building usable representations of the environment to reach goals, avoid obstacles, and operate in real-world settings. Prior visual navigation policies are often supervised, goal-conditioned models that can become difficult to adapt when new constraints appear, and they cannot easily allocate additional computation to harder planning problems. Existing world models and view synthesis methods also tend to focus on narrow settings or rely on structured 3D priors, limiting their scalability across diverse environments and embodiments. The authors address this gap by introducing a Navigation World Model trained on robot videos and navigation actions, along with a new Conditional Diffusion Transformer architecture that efficiently predicts future visual states and supports model-predictive planning and policy ranking, while also benefiting from unlabeled video data for improved generalization.

Method

The proposed system learns a navigation world model as a stochastic state transition function. The input consists of egocentric images and navigation commands. Each image observation is encoded into a latent state, while each action contains translation and yaw control signals. Formally, the model is trained from sequences of the form

where $x_i \in \mathbb{R}^{H \times W \times 3}$xi​∈RH×W×3 is an image and $a_i=(u,\phi)$ai​=(u,ϕ) is a navigation command, with $u \in \mathbb{R}^2$u∈R2 controlling forward and lateral motion and $\phi \in \mathbb{R}$ϕ∈R controlling yaw rotation. Images are mapped into compressed latent states using a pretrained VAE encoder:

The world model then predicts the next latent state conditioned on the current state, a short history of past states, and the action:

where $\mathbf{s}_{\tau}=(s_{\tau},\ldots,s_{\tau-m})$sτ​=(sτ​,…,sτ−m​). This formulation supports different environments and action spaces because the model operates on abstract latent states rather than environment specific representations.

To model temporal dynamics more explicitly, the action is extended with a time shift input $k$k. The action becomes

where $k$k specifies how many future or past steps should be traversed. The effective translation and rotation for the shifted interval are computed by aggregating the intermediate actions:

This allows the model to learn both control effects and passive temporal changes in the environment. During training, multiple possible goals are sampled from the same context to reduce entanglement between time and action.

The core predictor is implemented with a conditional diffusion transformer, which is trained as a denoising model over future latent states. The architecture uses a temporally autoregressive design. Tokens from the target frame being generated attend to one another through self-attention, while past frames provide contextual information through cross-attention. This keeps the main self-attention computation restricted to the target frame and uses previous latent states as keys and values for contextualization. Refer to the architecture diagram:

The conditioning signal combines the navigation action, the time shift, and the diffusion timestep. Each scalar component is first lifted through sine-cosine features and a small MLP, then concatenated or projected into the model dimension. The final conditioning vector is formed by summing the action, time shift, and timestep embeddings:

This vector is fed into an adaptive layer normalization block, which produces scale and shift parameters used to modulate the transformer sublayers. When navigation actions are unavailable, the action embedding can be omitted, allowing the same architecture to be trained on unlabeled video sequences.

The computational benefit of this design comes from its attention structure. A standard transformer over all frames has attention cost dominated by the square of the total context length. The proposed block instead uses target-frame self-attention and cross-attention to past frames, giving a cost that scales linearly with the number of context frames. If $n$n is the number of tokens per frame, $m$m is the number of frames, and $d$d is the token dimension, this reduces the dominant dependence from quadratic in the full sequence length to linear in the number of frames.

Training follows a standard diffusion process. In the forward process, Gaussian noise is added to the clean next latent state $s_{\tau+1}$sτ+1​ according to a randomly sampled diffusion timestep $t$t:

where $\{\alpha_t\}${αt​} is a noise schedule. The network is trained to recover the clean latent state from the noisy state, conditioned on the context, the action, and the diffusion timestep. The primary objective is a mean squared prediction loss:

The model can also predict the covariance of the diffusion noise and be trained with a variational lower bound term.

Once trained, the world model can be used for planning by rolling out candidate navigation actions and scoring the resulting future states. Given an initial latent state $s_0$s0​ and a target latent state $s^{*}$s∗, the objective is to find an action sequence that maximizes the likelihood of reaching the goal. This is expressed with an energy function that penalizes low similarity to the goal and invalid actions or unsafe states:

The goal similarity $\mathcal{S}(s_T,s^{*})$S(sT​,s∗) can be computed by decoding latent states back to pixel space with the VAE decoder and measuring perceptual similarity. Planning can then be performed as a model predictive control problem by minimizing this energy over future action sequences. In practice, derivative-free population optimization such as the cross entropy method can be used to search over actions. The same energy can also rank trajectories proposed by an existing navigation policy by selecting the sampled rollout with the lowest energy.

When the model is applied outside its training distribution, long horizon imagined trajectories may slowly drift away from the true visual context. The generated future frames can gradually become dominated by common training patterns rather than the current observation. This behavior is visible in generated future frames, as illustrated in the qualitative example:

Experiment

The experiments evaluate a navigation world model on multiple robot navigation datasets, focusing on future video prediction, generation quality, planning, and generalization to unseen environments through comparisons with prior baselines. Ablations show that using a context-conditioned model, multiple goals, sufficient temporal context, and both time and action inputs improves predictive accuracy. Further tests demonstrate that the model enables effective goal-conditioned and constrained planning, can improve trajectory ranking for existing navigation policies, and benefits from unlabeled videos for better transfer to unknown scenes, though with some increased hallucination.

For four-second future prediction on RECON, using more predicted goals and more conditioning context consistently improves predicted-frame quality. Predictions conditioned only on time are much worse than goal-conditioned settings, highlighting the importance of visual goal information. Four goals give better prediction than one or two goals across all reported metrics. Longer context helps prediction, while short context leads to poorer tracking.

The results compare navigation and video models on predicting future observations and synthesizing longer rollouts. The proposed NWM shows substantially better video synthesis quality than the diamond baseline, while the evaluation uses lower perceptual error to indicate better future prediction at a given computational budget. NWM greatly reduces FVD relative to the diamond baseline, suggesting more realistic generated videos. Future prediction quality is measured by perceptual error versus computational cost, where lower LPIPS indicates more accurate prediction.

NWM outperforms GNM and NoMaD on 2-second trajectory prediction for goal-conditioned visual navigation, achieving the best ATE and RPE among the reported methods. When NWM is used to rank NoMaD trajectories, it also improves both translation and rotation error, with more sampled trajectories giving better results. NWM planning yields the lowest ATE and RPE among the compared navigation approaches NWM-based trajectory ranking improves NoMaD performance on both ATE and RPE Using more NWM-ranked trajectories further improves navigation accuracy

Constrained planning remains close to the unconstrained baseline in both final position and yaw, indicating that the Navigation World Model can follow action restrictions without substantially degrading goal-directed behavior. The forward-first constraint produces the largest relative drift, while the left-right-first and straight-then-forward constraints show smaller changes. Overall, the results support effective planning when motion is restricted to predefined action patterns. All tested motion constraints are satisfied while final pose changes stay small relative to unconstrained planning. Forward-first planning shows the greatest sensitivity in yaw, whereas left-right-first has the smallest position drift. Straight-then-forward planning yields modest increases in both final position and yaw compared with the baseline.

Adding unlabeled Ego4D videos alongside in-domain data improves future prediction quality on unseen environments, with lower perceptual errors and higher PSNR on Go Stanford. The same change reduces prediction quality on the known RECON environment, where perceptual metrics worsen and PSNR declines. Overall, the result suggests unlabeled data helps generalization to new environments while introducing a trade-off on familiar data. Unlabeled Ego4D training improves unknown-environment metrics across LPIPS, DreamSim, and PSNR. Known-environment performance degrades after adding unlabeled data. The observed gain is modest but consistent on unseen data.

On RECON and goal-conditioned navigation benchmarks, experiments evaluate future-frame prediction, longer video rollouts, trajectory ranking, constrained planning, and cross-environment generalization. Goal conditioning, additional predicted goals, and longer context improve predicted-frame quality, while NWM outperforms baseline navigation and video models in future prediction, synthesizes more realistic rollouts, and improves NoMaD trajectory ranking. The model preserves near-unconstrained goal-directed behavior under action restrictions, and adding unlabeled Ego4D videos helps prediction in unseen environments at the cost of some known-environment accuracy.