One-sentence Summary

HKUST(GZ), Kling Team, PKU, and other institutions present KeyFrame-Compass, the first comprehensive benchmark for keyframe-conditioned video generation, which features 386 curated samples across diverse application domains, video structures, and keyframe densities, decomposes keyframe execution into six complementary metrics, and uses evidence-grounded MLLM judgments, revealing a trade-off between faithful keyframe reproduction and natural synthesis that worsens with denser constraints when evaluated on nine representative systems.

Key Contributions

• KeyFrame-Compass, the first comprehensive benchmark for keyframe-conditioned video generation, comprises 386 curated samples with systematically controlled factors including keyframe density, video structure, prompt granularity, input format, and application domain.
• The automated evaluation framework decomposes keyframe execution into six complementary metrics covering presence, appearance fidelity, temporal placement, ordering, persistence, and uniqueness, and combines them with evidence-grounded MLLM judgments augmented by specialized perception models.
• Stratified experiments on nine representative video generation systems reveal a trade-off between faithful keyframe execution and natural video synthesis, performance degradation under denser keyframe constraints, and failure of most open-source models to interpret storyboard-grid inputs as temporally ordered keyframe sequences.

Introduction

The authors address the growing need for keyframe-conditioned video generation, where models must take an ordered set of reference images and a prompt to produce a coherent video that preserves the exact appearance, order, and timing of each keyframe. This task sits at the heart of modern creative workflows, but existing benchmarks for general video generation evaluate only perceptual quality, temporal consistency, or single-image conditioning, none of which measure whether a full keyframe sequence is faithfully reproduced. The authors introduce KeyFrame-Compass, the first diagnostic benchmark specifically designed for multi-keyframe-conditioned video generation. It contains 386 curated cases across five controlled factors and two evaluation axes: six keyframe-response metrics (presence, fidelity, ordering, timing, persistence, and uniqueness) assess plan execution, while an evidence-grounded quality framework using multimodal large language models and perception models measures overall video quality. This separation reveals a persistent fidelity–naturalness trade-off and systematic failures as keyframe constraints become denser.

Dataset

The authors construct KeyFrame-Compass, a benchmark for evaluating keyframe-driven video generation.

• Dataset composition and sources

• The benchmark pulls narrative data from two origins: existing story datasets (ViStoryBench, VIST, ROCStories) and real videos paired with generated narrative captions.

• Both sources are unified into structured story summaries, which form the backbone of every test case.

• Key details and subsets

• The final dataset contains 386 test cases across three video structure/duration settings:

• One-take videos (5–10 s)

• Short multi-shot videos (5–10 s)

• Long multi-shot videos (10–45 s)

• Samples are distributed by the number of keyframes:

• 3 keyframes: 75 cases

• 6 keyframes: 112 cases

• 9 keyframes: 110 cases

• 12 keyframes: 89 cases

• Each story averages 3.73 unique characters, with most stories involving 3–5 characters.

• Scene annotations emphasize action, establishing, interaction, and payoff shots (67.2% of all labels). Dominant camera motions are truck/track, follow shot, pan, and push-in; frequent shot scales are medium, full, and wide.

• How the paper uses the data

• KeyFrame-Compass is a dedicated evaluation benchmark. The authors do not split it into training/validation sets; it serves purely to test video generation models.

• Every sample is delivered in two input formats with two prompt variants, plus complete metadata, enabling systematic assessment of keyframe alignment and narrative consistency.

• Processing and metadata construction

• Each story summary is expanded into a structured scene specification that includes narrative synopsis, video structure, subject descriptions, shot definitions, temporal layout, target keyframe positions, and cinematic annotations.

• Keyframe images are generated using GPT-Image-2 and Nano Banana Pro, then screened through multimodal consistency checks and human review to enforce visual quality, cross-frame subject consistency, and narrative relevance.

• The scene specification is then rewritten by Gemini 3.1 Pro into video-oriented prompts, preserving subject identity, spatial relations, and visual states. After quality review and safety screening, the final metadata is assembled.

• No cropping strategy is explicitly applied; the metadata captures camera motion and shot scale annotations to characterize cinematic diversity.

Method

The authors construct the KeyFrame-Compass dataset through a multi-stage pipeline encompassing raw data curation, specification conversion, spec-driven generation, and rigorous filtering.

Refer to the framework diagram.

The process begins with Raw Data Curation and Preprocessing, where the authors leverage two primary narrative sources. The first source comprises existing datasets, including ViStoryBench, VIST, and ROCStories, which provide storyboard references, text stories, and image sequences. The second source involves real-video transcription, where multimodal large language models generate narrative captions from source videos. Both streams undergo filtering for story information density, content safety, and sample diversity to produce structured story summaries.

In the Summary-to-Spec Conversion phase, these summaries are transformed into structured scene specifications using multimodal large language models. Each specification serves as a unified representation containing a story synopsis, video structure, subject definitions, keyframe positions, and cinematic annotations. This ensures consistent subject identity, scene layout, and temporal alignment throughout the sample.

The Spec-Driven Construction phase utilizes these specifications for two parallel tasks. For Keyframe Candidate Generation, temporal anchors are translated into image constraints covering subject appearance, action states, and spatial layout. Multiple candidate keyframes are generated using state-of-the-art text-to-image models, such as GPT-Image-2 and Nano Banana Pro. These candidates are then filtered through multimodal large language model evaluation and human review to verify cross-frame subject consistency and narrative relevance. Simultaneously, for Video Prompt Adaptation, the scene specifications are rewritten into video-oriented prompts. The authors generate two prompt levels: a minimal prompt providing global video-level instructions and a segment-specific prompt detailing shot-level sequences and generation goals.

Finally, the pipeline incorporates Quality Review and Safety Filtering. The authors evaluate samples for specification compliance, visual quality, and identity consistency. Safety screening is applied to avoid public figures, violence, sexual content, and child harm, resulting in a carefully curated dataset across various keyframe counts, domains, and structures.

Experiment

The KeyFrame-Compass benchmark evaluates video generation models by requiring them to faithfully reproduce keyframe images at specified temporal positions and synthesize coherent transitions across diverse prompt types and video structures. Results show that strong keyframe reproduction does not guarantee smooth motion, as LTX-2.3 excels at keyframe fidelity but suffers from abrupt transitions, while Gemini-Omni-Flash treats keyframes as loose semantic cues rather than visual anchors, often re-staging scenes. Seedance 2.0 achieves the best overall balance, and instruction adherence consistently declines as the density of visual constraints increases. Open-source models largely fail to comprehend storyboard grids, frequently generating static grid reproductions, and human evaluations confirm that the automatic metrics align well with human judgments.

Existing image-conditioned video generation benchmarks uniformly lack keyframe response metrics, meaning they do not evaluate whether models faithfully reproduce ordered keyframes at intended temporal positions. Only LongAV-Compass supports multi-granular prompt control, and most benchmarks rely on single-image conditioning, leaving a gap that KeyFrame-Compass fills by assessing keyframe adherence and temporal placement across varying numbers of keyframes. None of the compared benchmarks include keyframe response metrics, so temporal placement of prescribed keyframes is not evaluated. LongAV-Compass is the only benchmark that supports multi-granular prompt control, while all others use only single conditions (e.g., single image or not reported).

The KeyFrame-Compass benchmark contains 386 test cases distributed across three video structure-duration combinations (one-take 5-10s, multi-shot 5-10s, multi-shot 10-45s) and three application domains (daily, product, cinematic). Multi-shot videos account for the majority of samples, with the 5-10s multi-shot category being the largest, while the product domain is the least represented. Multi-shot videos of 5-10 seconds constitute the largest subset (166 cases), and one-take videos of the same duration are the smallest (61 cases). The daily domain dominates with 190 samples, while the product domain contributes only 53 cases, making them the most and least frequent domains respectively.

Seedance 2.0 leads the short-video leaderboard with an overall score of 0.807, combining balanced performance across all dimensions. Models that excel in one area, such as LTX-2.3 in keyframe fidelity or Gemini-Omni-Flash in video quality, fall behind in the joint evaluation, revealing rank reversals that a single-metric focus would miss. Seedance 2.0 ranks first in spatiotemporal coherence and instruction adherence, and second in all other dimensions, achieving the highest overall score. LTX-2.3 attains the best keyframe fidelity (0.855) and temporal organization (0.899) but drops to fourth overall due to a much lower video quality score (0.557).

Seedance 2.0 achieves the best overall performance by balancing keyframe control, video quality, and instruction adherence, while LTX-2.3 excels in keyframe fidelity but is hindered by lower video quality, and Gemini-Omni-Flash prioritizes video quality and audio-visual coordination at the expense of keyframe similarity. Increasing the number of keyframes reduces visual instruction adherence, and open-source models generally fail to interpret storyboard grids, instead reproducing the entire grid as a static video. Seedance 2.0 ranks first overall, leading in spatiotemporal coherence and instruction adherence while maintaining strong performance in all other dimensions. LTX-2.3 achieves the highest keyframe fidelity and temporal organization among all evaluated systems, but its lower general video quality results in a lower overall rank. Gemini-Omni-Flash has the highest video quality and audio-visual coordination scores, yet its keyframe fidelity is relatively low, and it often re-stages shots rather than preserving the visual content of input frames. All proprietary models show a drop in visual instruction adherence when moving from minimal to segment-specific prompts, indicating that denser visual constraints impair temporal and visual control. Open-source models that receive a storyboard grid often fail to decompose it, producing near-static videos of the grid, which suggests they have not learned storyboard comprehension.

Under segment-specific prompts, overall instruction adherence declines as the number of keyframes increases, falling from 0.849 with 3 keyframes to 0.756 with the merged 9&12 group. This drop is driven entirely by the visual and motion adherence (VMA) component, while audio adherence (AMA) remains nearly constant across all keyframe densities. Moving from 3 to 6 keyframes reduces the instruction score from 0.849 to 0.818, and further to 0.756 for 9&12 keyframes. VMA drops sharply from 0.861 (3 keyframes) to 0.682 (9&12 keyframes), while AMA stays within 0.830–0.837 across all groups. The degradation is concentrated in visual and temporal control; audio adherence is unaffected by the number of keyframes.

KeyFrame-Compass fills the gap in existing benchmarks by evaluating video generation models on keyframe adherence and temporal placement across 386 test cases that vary in duration, structure, and application domain. The evaluation shows that Seedance 2.0 leads overall by balancing keyframe fidelity, video quality, and instruction adherence, while other models trade off between these dimensions, such as LTX-2.3 excelling in keyframe fidelity but suffering from lower video quality. Increasing the number of keyframes consistently degrades visual and temporal instruction adherence, and open-source models often fail to interpret storyboard grids, instead producing static videos of the grid layout.