One-sentence Summary

Meta FAIR presents SAM 2, a foundation model for promptable visual segmentation in images and videos, built on a transformer architecture with streaming memory, a data engine, and the largest video segmentation dataset, achieving better video accuracy with $3 \times$3× fewer interactions and $6 \times$6× faster image segmentation than SAM.

Key Contributions

• The paper introduces SAM 2, a promptable foundation model for visual segmentation in both images and videos, built on a simple transformer architecture with streaming memory that enables real-time video processing.
• This work presents a data engine and the SA-V dataset, the largest video segmentation dataset to date, which extends beyond whole objects to extensively cover object parts and contains over an order of magnitude more masks than prior video segmentation datasets.
• Results demonstrate that SAM 2 achieves better video segmentation accuracy with 3× fewer interactions than prior approaches and is more accurate and 6× faster than SAM for image segmentation, with the model, dataset, interactive demo, and code publicly released.

Introduction

The authors tackle the limitation of image-only segmentation, which fails to address the temporal dimension required by applications in AR/VR, robotics, autonomous vehicles, and video editing. Prior video segmentation models and datasets struggle with appearance changes, occlusions, lower video quality, and lack of interactive refinement, and they do not achieve the generality of "segment anything in videos." As the main contribution, the authors introduce SAM 2, a unified model for promptable visual segmentation across images and videos, using a memory mechanism for temporal consistency, along with a new PVS task and the SA-V dataset containing 35.5M masks. SAM 2 delivers higher accuracy with 3 times fewer interactions and runs 6 times faster than SAM on images, while outperforming prior video segmentation benchmarks.

Dataset

The authors build a large-scale video segmentation dataset called SA-V to support the goal of "segment anything" in video. The dataset is collected through a data engine that combines an interactive model in the loop with human annotators. No semantic constraints are imposed on the annotated masklets, and annotations cover both whole objects and object parts.

Dataset Composition and Sources

• The SA-V dataset contains 50.9K videos with 642.6K masklets, released under a permissive license.
• Videos are newly collected by crowdworkers and include 54% indoor and 46% outdoor scenes, with an average duration of 14 seconds.
• Annotations comprise 190.9K manual masklets and 451.7K automatic masklets.
• The number of annotated masks is 53x larger than any existing VOS dataset (15x without automatic annotations).
• The dataset has a 42.5% disappearance rate in manual masklets, meaning masklets that disappear in at least one frame and then reappear, which is competitive with existing datasets.

Data Engine Phases

The authors use a three-phase data engine, with each phase increasing the level of model assistance for annotators.

• Phase 1 (SAM per frame): Annotators use the image-based interactive SAM to annotate masks in every frame at 6 FPS, using pixel-precise manual editing tools. There is no temporal propagation, leading to high quality per-frame annotations but slow throughput. This phase collects 16K masklets from 1.4K videos, with an average annotation time of 37.8 seconds per frame. Phase 1 is also used to annotate the SA-V val and test sets to avoid biases from SAM 2 during evaluation.
• Phase 2 (SAM + SAM 2 Mask): SAM 2 is added to the loop and accepts masks as prompts. Annotators first generate spatial masks in the first frame, then propagate them temporally. They can correct predictions in later frames and re-propagate. SAM 2 Mask is retrained twice during this phase. Phase 2 collects 63.5K masklets with annotation time reduced to 7.4 seconds per frame, a 5.1x speedup over Phase 1.
• Phase 3 (SAM 2): The fully-featured SAM 2 accepts points and masks as prompts and uses temporal memory to generate predictions. Annotators only need occasional refinement clicks instead of annotating from scratch. SAM 2 is retrained five times during Phase 3. This phase collects 197.0K masklets with annotation time at 4.5 seconds per frame, an 8.4x speedup over Phase 1, while maintaining comparable quality.

Quality Verification and Auto Masklet Generation

• A separate set of annotators verifies each masklet as "satisfactory" (correctly and consistently tracking the target object) or "unsatisfactory." Unsatisfactory masklets are sent back for refinement, and masklets tracking poorly defined objects are rejected.
• To increase annotation diversity, the authors generate automatic masklets by prompting SAM 2 with a regular grid of points in the first frame. Generated masklets are filtered through the verification step. Satisfactory automatic masklets are added to the dataset, while unsatisfactory ones are sampled and sent to annotators for refinement, which also helps identify model failure cases.

Dataset Splits and Internal Data

• The SA-V training, validation, and test splits are created based on video authors and their geographic locations to minimize overlap of similar objects.
• The val split has 155 videos and 293 masklets, and the test split has 150 videos and 278 masklets. Both are selected to focus on challenging scenarios, including fast-moving objects, complex occlusions, and disappearance or reappearance patterns.
• The authors also use an internal dataset with licensed video data, comprising 62.9K videos and 69.6K masklets annotated in Phase 2 and Phase 3 for training, plus 96 videos and 189 masklets annotated with Phase 1 for testing.

How the Data Is Used in the Model

• The data engine phases are used iteratively to retrain SAM 2, with the collected annotations from each phase added to the training set.
• The authors compare model performance after each phase, keeping training iterations fixed, and observe consistent improvements on both the in-domain SA-V val set and nine zero-shot benchmarks when prompting with three clicks on the first frame.

Method

The authors propose SAM 2, a model that generalizes the Segment Anything Model (SAM) to the video domain. The system processes video frames in a streaming fashion, allowing for real-time handling of arbitrarily long videos. The architecture integrates spatial segmentation capabilities with a temporal memory mechanism to maintain object consistency across frames.

The pipeline initiates with an image encoder that operates only once per interaction. The authors employ a hierarchical, MAE pre-trained Hiera encoder to generate unconditioned feature tokens for each frame. This hierarchical design facilitates the extraction of multiscale features necessary for decoding.

Central to the architecture is the memory attention module. This component conditions the current frame's features on memories from past predictions and prompts. The memory attention consists of stacked transformer blocks that perform self-attention followed by cross-attention to a memory bank. The memory bank retains information about past predictions using a FIFO queue structure, storing up to $N$N recent unprompted frame memories and up to $M$M prompted frame memories. These memories are maintained as spatial feature maps. Furthermore, the system stores lightweight object pointer vectors that capture high-level semantic information from the mask decoder output. The attention mechanism also embeds temporal position information into the memories of recent frames to represent short-term object motion.

Following the attention mechanism, the prompt encoder and mask decoder generate the segmentation output. The prompt encoder accepts points, boxes, or masks to define the spatial extent of the target object. Sparse prompts are encoded using positional encodings summed with learned embeddings, while masks are embedded via convolutions. The mask decoder utilizes "two-way" transformer blocks that iteratively update prompt and frame embeddings. To address the possibility of the target object being absent in certain frames, such as during occlusion, the decoder includes an additional head that predicts object presence. The design also incorporates skip connections from the hierarchical image encoder to provide high-resolution embeddings for mask decoding. In cases of ambiguity, the model predicts multiple masks and propagates the one with the highest predicted Intersection over Union (IoU) if subsequent prompts do not resolve the uncertainty.

After prediction, a memory encoder processes the result for future use. It downsamples the output mask via a convolutional module and fuses it with the unconditioned frame embedding from the image encoder using light-weight convolutional layers. This fused representation is then stored in the memory bank.

The model is trained jointly on image and video data. The training process simulates interactive prompting by sampling sequences of frames and randomly selecting frames for initial prompts. These prompts can be ground-truth masks, positive clicks, or bounding boxes. The training objective is to sequentially predict ground-truth masklets, allowing the model to learn from corrective clicks and refine its predictions iteratively.

Experiment

The experiments demonstrate that SAM 2 outperforms prior methods in zero-shot promptable video segmentation, achieving higher accuracy with over 3 times fewer interactions across multiple benchmarks in both offline and online settings. In zero-shot image segmentation, SAM 2 improves accuracy over SAM while running 6 times faster, with further gains from training on a combined image and video data mix. For semi-supervised video object segmentation, SAM 2 surpasses state-of-the-art approaches at real-time inference speeds, and larger image encoders consistently yield better accuracy, including on long-term and open-world video benchmarks.

The table compares three data annotation phases, showing progressive improvements in efficiency and quality as the model in the loop evolves. Phase 3, using the fully-featured SAM 2, achieves the fastest annotation time and lowest manual editing effort while maintaining or improving mask alignment accuracy across all mask sizes. Annotation time per frame drops from 37.8 seconds in Phase 1 to 7.4 seconds in Phase 2 and further to 4.5 seconds in Phase 3, an approximate 8.4x speedup. The percentage of edited frames per masklet declines sharply from 100% in Phase 1 to 23.25% in Phase 2 and 19.04% in Phase 3, indicating reduced manual intervention. Manual clicks per clicked frame decrease from 4.80 in Phase 1 to 3.61 in Phase 2 and 2.68 in Phase 3, while mask alignment to Phase 1 improves for all mask sizes, reaching 100% alignment for large masks in Phase 3.

Adding data from successive data engine phases progressively improves segmentation accuracy on both the SA-V val set and the combined zero-shot VOS plus SA-1B benchmark. The largest gains come from incorporating Phase 2 and Phase 3 data, with the fully interactive SAM 2 phase yielding near-final performance. Including additional automated data provides a small further boost. Moving from no phase data to Phase 1 data raises SA-V val accuracy by 3 points and the combined benchmark by 4.4 points. Phase 2 data adds 5.8 points on SA-V val and 4.0 points on the combined benchmark over Phase 1 alone. Phase 3 data contributes the largest per-phase gain on SA-V val, adding 3.7 points, while the combined benchmark improves by a further 0.3 points. Adding auto-generated data yields the final marginal improvement, reaching 63.2 on SA-V val and 71.5 on the combined benchmark.

The compared open-source video object segmentation datasets vary widely in scale, with internal data being the largest by a substantial margin in terms of videos, duration, and masks. Disappearance rates also differ considerably across datasets, with some having relatively low rates and others exceeding 40%. The internal dataset is orders of magnitude larger than all open-source alternatives in video count, total duration, and number of masks. Disappearance rates range from under 10% to over 40% across datasets, with VOST and MOSE showing the highest rates. Datasets with longer durations tend to have higher disappearance rates, but the internal dataset maintains a moderate rate despite its scale.

Across 17 video datasets, SAM 2 achieves higher zero-shot segmentation accuracy than SAM+XMem++ and SAM+Cutie for every prompt type, including 1, 3, and 5 clicks, bounding boxes, and ground-truth masks. Accuracy consistently improves with more clicks, and mask-based prompting yields the best results, with SAM 2 leading in all settings. SAM 2 outperforms both baselines across all prompt types, with the largest gains at higher interaction levels. Using ground-truth masks as prompts achieves the highest accuracy for all methods, and SAM 2 maintains the top performance in this setting.

SAM 2 exceeds SAM's average zero-shot accuracy on the original 23 datasets while running at over six times the speed, and further gains are achieved by training on a combined image and video data mix, particularly for zero-shot video datasets. SAM 2 with SA-1B data improves 1-click mIoU on the 23 standard datasets from 58.1 to 58.9. Training on the mixed data raises the 23-dataset average to 61.9 and lifts performance on the 14 additional video datasets from 56.6 to 69.6. SAM 2 runs at 130.1 FPS, roughly six times faster than SAM at 21.7 FPS.

The experiments evaluate an iterative data engine for training SAM 2, showing that each phase progressively reduces annotation effort and time while maintaining or improving mask alignment, with the full SAM 2 phase being the most efficient. Adding data from successive phases consistently boosts segmentation accuracy on both validation and zero-shot benchmarks, with the largest gains from the interactive phase. SAM 2 also outperforms prior baselines across diverse prompt types and datasets, while being significantly faster than SAM. Overall, the results confirm that scaling data through interactive model-in-the-loop phases is key to achieving strong, efficient video segmentation.