One-sentence Summary

VideoPoet is a decoder-only autoregressive Transformer trained via a two-stage LLM-style protocol of pretraining and task-specific adaptation over multimodal conditioning signals including images, video, text, and audio, achieving state-of-the-art zero-shot video generation with high-fidelity motion synthesis.

Key Contributions

• VideoPoet is a decoder-only Transformer trained as a large language model to synthesize high-quality videos from images, videos, text, and audio using autoregressive multimodal pretraining and task-specific adaptation.

• The method trains on tokenized text-paired and unpaired video data with a mixture of multimodal generative objectives, and it uses a bidirectional transformer with efficient windowed local attention for video super-resolution in latent token space.

• Evaluations and demonstrations show competitive and state-of-the-art zero-shot video generation, especially for high-fidelity, large, and complex motions across diverse video generation and editing tasks.

Introduction

Video generation has become a fast-growing application of foundation models, with most systems relying on diffusion methods that start from strong image generation models and add temporal modeling for video. This matters because high-quality video generation is important for creative, media, and multimodal AI systems, and language models offer a potential path to unified architectures that reuse mature LLM training and inference infrastructure. The authors note, however, that prior language-model-based video generators have not matched the visual quality of diffusion models, and that diffusion-based approaches often require separate adapters, architectural changes, or inference tricks to support multiple tasks and modalities.

The authors introduce VideoPoet, a decoder-only large language model for video generation that represents images, videos, and audio as discrete tokens. The model is pretrained on a mixture of multimodal objectives and then adapted to tasks such as text-to-video generation, image-to-video generation, video editing, and video-to-video stylization within a single architecture. Their contribution is a proof of concept that LLMs can be trained as versatile video generators, including methods for multi-task pretraining, autoregressive extension to longer videos, and a video super-resolution method that operates in the latent token space.

Dataset

• The authors use a large multimodal dataset totaling roughly 2 trillion tokens across modalities, built from public internet content and other sources.

• The core training data includes:

  • 1 billion image-text pairs.
  • About 270 million videos, of which:
    • About 100 million videos have paired text.
    • About 50 million of those text-paired videos are used for high-quality finetuning.
    • About 170 million videos have paired audio.

• Before training, the dataset is filtered to remove egregious content and sampled to improve contextual and demographic diversity.

• The model is trained on a mixture of pretraining tasks using this large multimodal corpus.

• For text-to-video evaluation, the authors also finetune on a high-quality training subset. Unless otherwise stated, they do not finetune on task-specific evaluation data.

• The evaluation protocol is zero-shot, meaning the model is not trained on the training portions of the target benchmarks.

• The evaluation data includes:

  • MSR-VTT and UCF-101 for text-to-video generation.
  • Kinetics 600 for frame prediction, where the first 5 frames are used as conditioning to predict the next 11 frames.
  • Something-Something V2 for inpainting and outpainting tasks.

Method

The authors leverage a token-based multimodal generation framework that converts visual and audio signals into discrete sequences and uses a large language-model backbone to perform video generation, audio generation, and related tasks in a unified latent space. The method consists of three main stages. First, modality-specific tokenizers map pixels, video frames, and audio waveforms into discrete tokens. Second, a prefix language-model backbone consumes these tokens together with text embeddings and autoregressively predicts the target visual and audio tokens. Third, a specialized super-resolution module increases video resolution and refines visual details while remaining in token space.

Refer to the framework diagram for the input-output sequence layout.

The layout separates bidirectional prefix inputs from autoregressively generated outputs. The prefix may contain text embeddings, visual conditioning tokens, audio conditioning tokens, and special tokens that mark modality boundaries and task identity. The language model then generates the output portion causally, producing visual tokens and audio tokens that are subsequently decoded by modality-specific decoders. This design allows a single autoregressive backbone to support many task combinations by changing the prefix pattern and the target output type.

For visual inputs, the model uses MAGVIT-v2 to tokenize images and videos. A video clip is encoded and quantized into an integer sequence. A 17-frame clip at $128 \times 128$128×128 resolution and 8 frames per second produces a latent shape of $(5,16,16)$(5,16,16), which is flattened into 1280 tokens with a vocabulary size of $2^{18}$218. Portrait videos at $128 \times 224$128×224 resolution produce a latent shape of $(5,28,16)$(5,28,16), corresponding to 2240 tokens. The tokenizer enforces causal temporal dependency, which supports generation of longer videos. Images are represented as a single-frame case with a consistent token shape of $(1,16,16)$(1,16,16). For inpainting and outpainting, the authors use a COMMIT encoding scheme.

For audio, a pretrained SoundStream tokenizer maps an audio segment to discrete tokens. A 2.125-second clip is embedded into 106 latent frames using a residual vector quantizer with four levels. The authors predict audio tokens level by level from lower to higher residual quantization levels. Each level has a disjoint vocabulary of 1024 codes, yielding a combined audio vocabulary of 4096 codes. Text is not discretized into the same token vocabulary; instead, pretrained T5 XL embeddings represent text guidance. These embeddings are projected into the transformer embedding space through a linear layer before being used as conditioning signals.

The central generator is a prefix language model with a decoder-only architecture. The model constructs task-specific sequences by arranging text embeddings, visual tokens, audio tokens, and special tokens into a prefix, then predicting visual and audio tokens in the output region. Training loss is applied only to the generated output portion. A task-specific token indicates the desired output modality or task type. Tasks that do not require a certain modality simply omit the corresponding input or output tokens, shortening the sequence. The same task token can be reused for different input variants when the desired output is identical, allowing the model to adapt based on the available conditioning signals.

The task prompt design supports many generation settings, including unconditioned video generation, text-to-video, video future prediction, image-to-video, video inpainting and outpainting, video stylization, audio-to-video, video-to-audio, and audio-video continuation. Video tokens are generated at two spatial resolutions, $128 \times 128$128×128 and $128 \times 224$128×224, and at two temporal lengths, 17 frames and 41 frames, both encoded at 8 frames per second. Conditioning tokens signal the desired resolution and duration. Images are treated as a special case of one-frame video at $128 \times 128$128×128 resolution. For text-to-image pretraining, the model can omit end-of-sequence and end-of-visual-output tokens, allowing continuous token generation and making image generation more consistent with longer video generation. For stylization, the model can condition on text, optical flow, and depth, where text controls appearance while flow and depth provide structural guidance.

The training strategy is designed to handle highly variable sequence lengths across tasks. The authors use Alternating Gradient Descent by grouping tasks according to sequence length and alternately sampling one group at each training iteration. This reduces padding overhead compared with direct packing. The pretraining schedule also separates image and video data. During the first portion of training, image data is sampled with much higher frequency, improving object-level understanding. Later in training, the sampling distribution shifts toward video data, allowing the model to focus on motion and temporal consistency. The pretrained model is then fine-tuned on higher-quality data for specific tasks. This fine-tuning stage helps reduce repetitive token predictions, improves output diversity, and supports stronger classifier-free guidance.

Generating high-resolution video directly through autoregressive language modeling is impractical because token counts grow rapidly. For example, a $17 \times 896 \times 512$17×896×512 video can produce a sequence of about $35{,}840$35,840 visual tokens, making autoregressive sampling computationally expensive. To address this, the authors train a non-autoregressive video super-resolution transformer that operates in token space. Instead of performing global self-attention over the full high-resolution sequence, the module uses windowed local attention. Transformer blocks contain three attention layers, each attending over a local window aligned with temporal, spatial-horizontal, or spatial-vertical dimensions. Cross-attention layers attend to low-resolution token sequences and are similarly restricted to local windows. The module also attends to T5 XL text embeddings.

Refer to the architecture diagram below for the custom transformer design used in the super-resolution module.

To manage the large discrete vocabulary, the super-resolution transformer uses token factorization with $k=2$k=2 factors, converting a $262{,}144$262,144-way classification problem into two $512$512-way classification problems. Low-resolution conditioning sequences are constructed by tokenizing bicubic-downsampled ground-truth videos. Noise augmentation in the discrete latent space helps reduce the mismatch between real low-resolution videos and generated low-resolution videos. During training, a random subset of low-resolution token values is resampled, and the low-resolution condition and text embedding are independently dropped for a fraction of training samples. During inference, the module uses non-autoregressive sampling with classifier-free guidance applied separately to the low-resolution condition and the text condition. The system stacks two $2\times$2× super-resolution stages to produce videos at $896 \times 512$896×512 resolution from the base $224 \times 128$224×128 output of the language-model backbone.

Experiment

The evaluation uses zero-shot generation on text-to-video, frame-prediction, and inpainting or outpainting benchmarks, with separate finetuned text-to-video and human preference comparisons. Ablations show that combining text-paired video and image tasks with self-supervised tasks gives the most balanced overall performance, while scaling improves capability. Comparisons with recent diffusion and token-based models indicate competitive or superior performance, especially on motion interestingness and realism, and the model exhibits unified capabilities such as image-to-video, long-video generation, editing, stylization, and camera-motion control, although token reconstruction limits fine detail and visual fidelity.

Comparable 300M models show that text-paired data is important for zero-shot text-to-video generation, while self-supervised-only training remains weak on text-conditioned quality. Adding image-text pretraining improves text-video alignment and can make painting metrics better than text-to-video alone. The full multi-task setup achieves the best average balance across diverse evaluations, even though some narrower task subsets can be stronger on individual metrics. Text-to-video generation becomes weaker in text-video alignment when image-text pretraining is absent Self-supervised-only models can improve some unconditional and frame prediction metrics but fail to provide strong zero-shot text-to-video evaluation Omitting image-text pretraining gives the strongest painting results, while the full multi-task model provides a more balanced overall profile

The table compares zero-shot text-to-video results on MSR-VTT and UCF-101 using CLIP similarity and FVD scores. The evaluated model performs favorably against prior baselines, especially in CLIP similarity and FVD. The pretrained foundation model is competitive across metrics, and finetuning on a high-quality text-video subset further improves CLIP similarity on MSR-VTT. The model shows favorable zero-shot text-to-video performance on MSR-VTT and UCF-101. Finetuning on high-quality text-video pairs improves CLIP similarity on MSR-VTT.

Experiments comparing training data strategies show that text-paired data is essential for zero-shot text-to-video generation, while self-supervised-only training remains weak on text-conditioned quality. Image-text pretraining improves text-video alignment and can strengthen painting-related results, and the full multi-task setup provides the most balanced overall performance across diverse evaluations. Additional comparisons on MSR-VTT and UCF-101 indicate competitive zero-shot text-to-video results, with finetuning on high-quality text-video data further improving text-video similarity.