ClimbMix Dataset is a high-quality dataset released by NVIDIA in 2025, specifically designed for efficient pre-training of large language models. The related research paper is "CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training", aiming to provide pre-training performance superior to existing public datasets under the same token budget through innovative clustering and iterative mixing algorithms.

The dataset contains 400 billion tokens, stored in text format (Parquet), and is preprocessed and tokenized using the GPT-2 tokenizer. The dataset construction process first divides the raw data into 1,000 groups based on topic information, then applies two types of classifiers—advertising detection and educational value assessment—to score each group, removing low-quality data, and finally mixes the remaining high-quality data groups according to specific weights. The dataset is intended for research and development purposes only, helping researchers explore more efficient model scaling trends.

Dataset Composition

The dataset mainly includes the following key information:

• Data Scale: 400 billion tokens
• Data Format: Text sequences stored in Parquet format
• Tokenization: Tokenized using the GPT-2 tokenizer
• Data Source: Automated collection and processing
• Labeling Method: Automated labeling and filtering

Citation

@article{diao2025climb,
  author    = {Shizhe Diao and Yu Yang and Yonggan Fu and Xin Dong and Dan Su and Markus Kliegl and Zijia Chen and Peter Belcak and Yoshi Suhara and Hongxu Yin and Mostofa Patwary and Celine Lin and Jan Kautz and Pavlo Molchanov},
  title={CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training}, 
  journal   = {arXiv preprint},
  year      = {2025},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url={https://arxiv.org/abs/2504.13161}, 
}