Abstract

This paper presents PaLI-3, a smaller, faster, and stronger vision languagemodel (VLM) that compares favorably to similar models that are 10x larger. Aspart of arriving at this strong performance, we compare Vision Transformer(ViT) models pretrained using classification objectives to contrastively(SigLIP) pretrained ones. We find that, while slightly underperforming onstandard image classification benchmarks, SigLIP-based PaLI shows superiorperformance across various multimodal benchmarks, especially on localizationand visually-situated text understanding. We scale the SigLIP image encoder upto 2 billion parameters, and achieves a new state-of-the-art on multilingualcross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindlesresearch on fundamental pieces of complex VLMs, and could fuel a new generationof scaled-up models.