DataCompDR-12M is an image-text dataset released by Apple in 2024 that includes synthetic text, image embeddings, and metadata. Its associated paper can be found at 「MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training, aiming to improve the training efficiency and performance of image-text models through multimodal data augmentation strategies.

The dataset comprises metadata for 12.8 million images, generated based on a subset of DataComp-1B. It utilizes the coca_ViT-L-14 model from OpenCLIP to generate five pieces of synthetic text per image, combined with strong random image augmentation techniques. The dataset contains image embeddings, text embeddings, parameterized augmentation information, along with unique identifiers and hash values for each image, making it suitable for training efficient image-text alignment models.

Dataset Composition

• url.txt: Image URLs (string type)
• syn.json: Contains a list of synthetic texts (syn_text), which serve as alternative descriptions generated by the model
• paug.json: Contains a list of parametric augmentations (param_aug), recording specific parameters used for image enhancement
• .npz files: Contain image embeddings (image_emb) and text embeddings (text_emb), both represented as lists of floating-point numbers
• .json files: Include unique identifiers for images (uid) and their corresponding SHA256 hashes (sha256)

Each data sample consists of a triplet comprising randomly augmented images, real text, and synthetically selected text. The embedding vectors have a dimensionality of 1,536, formed by concatenating two 768-dimensional vectors derived from powerful teacher models.

Dataset Examples

Citation

@InProceedings{mobileclip2024,
  author = {Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, Oncel Tuzel},
  title = {MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2024},
}