Nemotron Content Safety Dataset V2 is a dataset released by NVIDIA in 2025, focusing on content safety and risk assessment for large language models, designed to provide high-quality multi-turn dialogue annotation data for training robust AI safety guardrail models.

The dataset contains 33,416 human-machine interaction samples, including 30,007 training samples, 1,445 validation samples, and 1,964 test samples. The dataset follows a comprehensive risk taxonomy consisting of 12 top-level hazard categories and 9 fine-grained subcategories, suitable for building content moderation and alignment systems for LLMs. The data is based on Anthropic HH-RLHF prompts, with responses generated using the Mistral-7B-v0.1 model, and combines human annotation with a multi-LLM jury system for safety classification.

Dataset Composition

• id: Identifier for the sample.
• prompt: The user's first-turn conversation. If derived from the Suicide Detection dataset and desensitized, it appears as "REDACTED" here, linked to the original data via reconstruction_id_if_redacted.
• response: The assistant's first-turn reply. May be null or an empty string for prompt-only samples.
• prompt_label: Binary safety label for the prompt.
• response_label: Binary safety label for the response. Null for prompt-only samples.
• prompt_label_source: Annotation source, always human.
• response_label_source: Response annotation source, including human, llm_jury, or refusal_data_augmentation.
• violated_categories: Comma-separated list of hazard categories, sorted by annotation frequency.

Citation```bibtex

@inproceedings{ghosh-etal-2025-aegis2, title = "{AEGIS}2.0: A Diverse {AI} Safety Dataset and Risks Taxonomy for Alignment of {LLM} Guardrails", author = "Ghosh, Shaona and Varshney, Prasoon and Sreedhar, Makesh Narsimhan and Padmakumar, Aishwarya and Rebedea, Traian and Varghese, Jibin Rajan and Parisien, Christopher", editor = "Chiruzzo, Luis and Ritter, Alan and Wang, Lu", booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)", month = apr, year = "2025", address = "Albuquerque, New Mexico", publisher = "Association for Computational Linguistics", url = "https://aclanthology.org/2025.naacl-long.306/", doi = "10.18653/v1/2025.naacl-long.306", pages = "5992--6026", ISBN = "979-8-89176-189-6" }