Abstract

Large multimodal models trained on natural documents, which interleave imagesand text, outperform models trained on image-text pairs on various multimodalbenchmarks. However, the datasets used to train these models have not beenreleased, and the collection process has not been fully specified. We introducethe OBELICS dataset, an open web-scale filtered dataset of interleavedimage-text documents comprising 141 million web pages extracted from CommonCrawl, 353 million associated images, and 115 billion text tokens. We describethe dataset creation process, present comprehensive filtering rules, andprovide an analysis of the dataset's content. To show the viability of OBELICS,we train vision and language models of 9 and 80 billion parameters namedIDEFICS, and obtain competitive performance on different multimodal benchmarks.We release our dataset, models and code.