ClimbLab is a high-quality pre-trained language model dataset released by NVIDIA in 2025, with the related paper titled "CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training", designed to provide large language models with a training dataset that has undergone strict filtering and clustering optimization.

The dataset is a filtered corpus containing 1.2 trillion tokens, divided into 20 semantic clusters. It is primarily intended for research and development purposes, formatted as Parquet, with a data scale of approximately 400 billion tokens (the official pre-tokenized version), and is collected and annotated through automated methods. Based on Nemotron-CC and SmolLM-Corpus, it uses the CLIMB clustering method to perform semantic reorganization and filtering of the data. The specific process includes dividing the data into 1,000 groups by topic and scoring them using two classifiers—advertising detection and educational value assessment—to remove low-quality data.

Dataset Composition

The structure and composition of the ClimbLab dataset are as follows:

• Data Format: Parquet format, containing token sequences tokenized using the GPT-2 tokenizer. To obtain the original text, the official detokenize_climblab.py script can be used for detokenization.
• Data Clustering: The data is divided into 20 distinct clusters, organized based on semantic relevance, which helps improve pre-training efficiency.
• Data Scale: The original corpus is 1.2 trillion tokens, while the official released pre-tokenized dataset version is approximately 400 billion tokens.
• Content Features: After advertising filtering and educational value assessment, the data ensures high quality and educational relevance.

Citation

@article{diao2025climb,
  author    = {Shizhe Diao and Yu Yang and Yonggan Fu and Xin Dong and Dan Su and Markus Kliegl and Zijia Chen and Peter Belcak and Yoshi Suhara and Hongxu Yin and Mostofa Patwary and Celine Lin and Jan Kautz and Pavlo Molchanov},
  title={CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training}, 
  journal   = {arXiv preprint},
  year      = {2025},
  archivePrefix = {arXiv},
  primaryClass = {cs.CL},
  url={https://arxiv.org/abs/2504.13161}, 
}