Hugo Laurençon Léo Tronchon Matthieu Cord Victor Sanh

Abstract

The growing interest in vision-language models (VLMs) has been driven byimprovements in large language models and vision transformers. Despite theabundance of literature on this subject, we observe that critical decisionsregarding the design of VLMs are often not justified. We argue that theseunsupported decisions impede progress in the field by making it difficult toidentify which choices improve model performance. To address this issue, weconduct extensive experiments around pre-trained models, architecture choice,data, and training methods. Our consolidation of findings includes thedevelopment of Idefics2, an efficient foundational VLM of 8 billion parameters.Idefics2 achieves state-of-the-art performance within its size category acrossvarious multimodal benchmarks, and is often on par with models four times itssize. We release the model (base, instructed, and chat) along with the datasetscreated for its training.