One-sentence Summary

The GigaBrain Team introduces GigaBrain-0.7, an embodied foundation model with a three-system architecture that unifies understanding, prediction, and action, scaling pretraining to over 37,000 hours of heterogeneous embodied data and employing one-stage alignment training to jointly optimize vision-language understanding and multi-embodiment action generation, thereby outperforming $\pi_{0.5}$π0.5​ and prior state-of-the-art models in foundation zero-shot capabilities, language-conditioned instruction following, and task success across both home and industrial scenarios.

Key Contributions

• GigaBrain-0.7 is introduced as a three-system architecture that coordinates vision-language understanding, future-state prediction, and multi-embodiment action generation. System 3 is separately pretrained for future-state prediction and task-progress estimation, and its predicted subgoal images and value-derived progress signals condition task-specific post-training and inference.
• Pretraining is scaled to over 37,000 hours of heterogeneous embodied data spanning 16 robot morphologies, combined with large-scale vision-language supervision.
• A one-stage alignment training procedure jointly optimizes multimodal understanding, hierarchical task prediction, discrete action supervision, and continuous action generation across heterogeneous embodiments.

Introduction

Vision-language-action (VLA) models adapt pretrained vision-language models for robot control, but scaling to diverse embodiments and action spaces often introduces interference rather than transferable knowledge. Most VLAs remain reactive, lacking mechanisms to anticipate future states or evaluate task progress, and they treat understanding, prediction, and action as separate rather than coordinated processes. The authors present GigaBrain-0.7, an embodied foundation model that unifies heterogeneous multi-embodiment pretraining with a three-system architecture: System 1 for continuous action generation, System 2 for hierarchical understanding and planning, and System 3 for future-state prediction and value estimation. Pretrained on over 37,000 hours of robot data across 16 morphologies and 270 million vision-language samples, GigaBrain-0.7 jointly optimizes multimodal understanding and action in one stage, then uses predicted subgoals and progress signals to condition policy post-training and enable experience-driven improvement.

Dataset

The authors construct a multi-source training corpus for a Vision-Language-Action (VLA) model, combining embodied trajectory data and vision-language data to provide complementary supervision during pretraining.

• Embodied trajectory data (37,256.98 hours after cleaning)

  • Real-robot data (20,535.65 h): 1,810,101 episodes from 16 robot types (e.g., Maker H01, Maker M01, Agibot-G1, AgileX, Franka, UR5). Provides primary continuous-action supervision.
  • UMI demonstrations (8,251.83 h): In-house Maker U01 and Jianzhi 10K. Offers near-manipulation viewpoints for hand-eye coordination and local spatial geometry.
  • EGO human demonstrations (2,862.36 h): Maker E01, EgoDex, EgoVerse, WiYH. Supplies first-person manipulation priors and broader object interaction categories.
  • Simulation data (1,453.92 h): Physics-based environments with configurable tasks and scenes, targeting configurations costly or unsafe to collect physically.
  • World-model-generated data (4,153.22 h): Prompt-conditioned video generation from curated real interactions, expanding visual and scene diversity while preserving interaction context.

• Vision-language data (271,976,674 image-text/QA samples, including 15,234,327 self-collected) Covers image captioning, general VQA, multi-image reasoning, region grounding, point prediction, spatial-relation reasoning, affordance understanding, and robotic task understanding. Preserves foundational vision-language capabilities and strengthens spatial, localization, and affordance reasoning for embodied tasks.

Processing pipeline

• Unified embodied format: All trajectory sources are converted to LeRobot v3.0 with standardized metadata, robot state/action representations (ordered by left arm, right arm, head, waist, base/legs), and camera definitions. Missing motion components use masks or placeholders.
• Instruction rewriting: An LLM (GLM-5.1) normalizes raw instructions into canonical task descriptions; a multimodal model adds subtask-level annotations for long-horizon tasks.
• Multi-stage cleaning:
  • q01/q99 extreme-value filtering per robot type and action dimension to avoid normalization instability.
  • Removal of prolonged stationary segments using inter-frame L2 change magnitude.
  • End-effector pose consistency correction via URDF alignment and reference point configuration.
  • Anomalous training loss filtering: trajectories with persistently high losses are traced back and removed if corrupted or misaligned.
• UMI-specific processing: Multi-camera synchronization, quality screening for smoothness and cross-modal consistency, retargeting to a robot TCP frame, and executability validation via inverse kinematics and real-robot replay.
• EGO-specific processing: Video-level filtering for clear hand-object interactions; standardization of timestamps, camera parameters, and hand trajectories; annotation of hand positions, object contacts, and spatial relationships using pose estimation or existing labels.
• Simulation/generated data: Physics-based simulation runs configurable tasks; world-model generation uses prompt-conditioned video expansion from real interactions. Both pipelines broaden long-tail coverage.
• VLM data processing: All sources are unified into a multimodal dialogue format with image quality control and structural validation. Embodied VLM annotations are generated from trajectory keyframes using Qwen3.6-27B, producing supervision for affordance prediction, spatial understanding, and high-level task planning. Automatic and human spot-check quality control filters low-confidence or ambiguous samples.

Usage in training The two data families are used jointly during pretraining. Embodied trajectories supply state-action supervision and interaction dynamics, while VLM data preserves general visual-semantic capabilities and injects spatial, grounding, and affordance reasoning. The corpus does not specify explicit mixture ratios; both components are combined to train the VLA model without degrading vision-language performance from action-only data.

Method

The authors design GigaBrain-0.7 as a comprehensive vision-language-action framework that bridges semantic understanding with continuous robot control. To address the complexities of long-horizon physical interaction, the architecture is organized into three progressively coupled systems that handle understanding, prediction, and action generation.

As shown in the figure below:

System 2 serves as the understanding and planning module. It operates as a PaliGemma2 (3B) vision-language model that interprets the current visual observation alongside the task instruction. By generating chain-of-thought reasoning, it identifies relevant objects and their spatial arrangements, subsequently decomposing the high-level task into immediate subtask instructions. This decomposition provides a crucial intermediate interface between high-level semantic reasoning and low-level motor control.

System 3 functions as the prediction and evaluation component, built upon a GigaWorld-1 (5B) world value model. It receives the current embodied context and the subtask description from System 2 to model future task evolution. This system exposes two complementary signals. First, it generates a short future video aligned with the subtask horizon, extracting the final frame as a subgoal image that represents the anticipated physical state. Second, it estimates a scalar value reflecting the current task progress. This value is converted into a binary advantage condition indicating whether the task progress is increasing, providing a compact evaluation of the execution state. The advantage condition is discretized as:

System 1 acts as the action and control core, integrating multimodal context into executable robot commands. It extends a PaliGemma2 (3B) vision-language backbone with a dedicated Action Expert (0.5B) using a Mixture-of-Transformers design. Let $H_{l}^{VL}$HlVL​ and $H_{l}^{AE}$HlAE​ denote the hidden states of the vision-language and action-expert streams at layer $l$l. The streams are coupled as follows:

where $\mathrm{Attn}(\mathbf{Q};\mathbf{C})$Attn(Q;C) denotes attention with queries from $\mathbf{Q}$Q and keys/values from context $\mathbf{C}$C. The vision-language stream employs causal self-attention to preserve the pretrained autoregressive interface, whereas the action expert attends bidirectionally to the union of both streams for grounded action generation. To balance pretrained representations with robot-specific learning, the authors introduce Soft Knowledge Insulation. This mechanism attenuates the gradients from the continuous action pathway before they enter the vision-language backbone:

where $0 < \alpha_{\mathrm{KI}} < 1$0<αKI​<1. This preserves general capabilities while adapting to embodied control.

To handle temporal dependencies, System 1 incorporates short-term visual memory via Temporal-Spatial Blocks in the visual encoder. These blocks factorize visual aggregation into temporal and spatial processing, fusing historical information into the current-frame representation without increasing the context length. Furthermore, an embodiment-aware state interface projects proprioceptive states and Robot IDs into the shared representation space, utilizing validity masks to handle heterogeneous robot morphologies. System 1 supports both a discrete next-token prediction pathway for symbolic actions and a continuous flow-matching pathway for precise trajectory generation.

The training process unfolds across four distinct stages. In the VLA pre-training stage, Systems 1 and 2 are jointly optimized on a heterogeneous corpus. The authors simultaneously optimize the autoregressive next-token prediction objective and the flow-matching objective for continuous actions. Temporal context is supervised by randomly dropping historical observations, and hierarchical task supervision couples high-level reasoning with low-level control.

During the world model pre-training stage, System 3 is developed through robot-centric video pre-training followed by value learning. The model learns to generate future observations and estimate task progress based on trajectory-level completion annotations. Once trained, the parameters of System 3 are frozen.

In the world-model-powered VLA post-training stage, the frozen System 3 provides the subgoal image and value-derived advantage condition to System 1. The authors apply stochastic condition dropout to prevent over-reliance on these predictive signals. The policy is then optimized using the flow-matching objective under the augmented context, allowing the model to learn how predictive signals influence task-specific action generation.

Finally, the experience-driven reinforcement learning stage refines the policy using its own deployment experience. The authors first conduct offline experience reinforcement using an Advantage-Weighted Regression objective to up-weight successful rollout segments. This is followed by online reinforcement with human correction, where an actor-critic framework is employed. Human operators intervene during difficult states to provide corrective demonstrations, and the policy is updated iteratively through a continuous rollout-correction-update cycle.

Experiment

The experiments evaluate GigaBrain-0.7 on real-robot platforms including AgileX PiPER/PiPER-X and the Maker H01 humanoid, along with embodied vision-language and simulation benchmarks. Architecture comparisons validate the PaliGemma2 dual-stream System 1 design, while scaling studies show that larger and more diverse pretraining data, especially with added human demonstrations, improve complex manipulation. Temporal context resolves ambiguous repeated states, and System 3's future-state and value conditioning improve robustness and execution efficiency. The model shows out-of-the-box language following and complex manipulation with partial out-of-distribution generalization, improves after task-specific post-training, and gains further from staged offline and online reinforcement learning.

The cleaned embodied trajectory corpus totals 37,257 hours, with real-robot data forming the largest share at 55.1%. UMI demonstrations contribute 22.2%, while egocentric human demonstrations, world-model-generated data, and simulation together account for the remaining 22.7%, providing a multi-source foundation for robot-control supervision. Real-robot trajectories dominate the corpus, making up over half of the total duration. UMI demonstrations are the second largest component, contributing roughly one-fifth of all hours. Simulation data represents the smallest fraction, accounting for less than 4% of the corpus. World-model-generated data supplies more hours than egocentric human demonstrations, at 11.2% versus 7.7%.

Among the listed robot types, real-robot data are highly concentrated, with Maker H01 supplying the largest share of episodes and frames. Agibot-G1 and AgileX follow as substantial secondary sources, while Ark, Galaxea R1 Lite, Franka, and UR5 each account for only a small fraction. The distribution reflects a few dominant platforms alongside limited cross-embodiment coverage. Maker H01 is the largest contributor by both episode count and frame proportion, approaching 44 percent of real-robot frames. Agibot-G1 and AgileX form the next largest tier, while the remaining listed platforms each contribute only modest shares.

The UMI dataset contributes 8,251.83 hours of manipulation demonstrations, with in-house Maker U01 providing the vast majority. The EGO dataset adds 2,862.36 hours of egocentric human demonstrations across four sources, led by In-house Maker E01 and EgoVerse. These collections supply diverse visual-action supervision for embodied learning. UMI demonstrations total 8,251.83 hours, with in-house Maker U01 accounting for 6,876.25 hours and Jianzhi 10K for 1,375.58 hours. EGO data spans 2,862.36 hours, where In-house Maker E01 (1,136.37 h) and EgoVerse (835.77 h) are the largest components.

Across the compared vision-language model backbones, larger model size did not consistently improve real-robot task success. Gemma 4 performed strongly on structured tasks such as clean desk and fruit picking, while PaliGemma2 was the only candidate that achieved nonzero success on shirt folding and remained competitive on fruit manipulation. Gemma 4 has the highest clean desk and fruit picking success among the compared backbones, but its clean desk result comes from a limited-range position-generalization setting. PaliGemma2 is the only backbone that achieves nonzero success on shirt folding while remaining competitive on fruit manipulation. Qwen3.5 performs relatively well on clean desk but drops sharply on fruit picking and records no successes on shirt folding.

With a fixed PaliGemma2 backbone, dual-stream coupling between the vision-language model and action expert produces the strongest real-robot results across all three tasks, including the only nonzero shirt-folding success. Last-layer and multi-layer cross-attention variants are cheaper at inference but much weaker on clean desk and fruit picking, and both fail on shirt folding. Multi-layer cross attention also increases training cost without improving performance over the last-layer variant. Dual-stream coupling achieves the highest success rates across clean desk, fruit picking, and shirt folding. Cross-attention variants reduce inference cost relative to dual-stream but perform substantially worse on every task and achieve zero shirt-folding success. Multi-layer cross attention has higher training cost than last-layer cross attention without yielding better task performance.

The experiments evaluate a large multi-source embodied trajectory corpus and several real-robot policy configurations. The corpus is dominated by real-robot data and concentrated on a few platforms, with UMI and egocentric human demonstrations providing additional visual-action supervision. In real-robot task evaluations, larger vision-language backbones do not consistently improve success, PaliGemma2 is the only backbone with nonzero shirt folding, and dual-stream coupling between the vision-language model and action expert is strongest across tasks, while cheaper cross-attention variants perform much worse.