SocialMaze is a social reasoning benchmark dataset that focuses on hidden role reasoning tasks in multi-agent interaction scenarios. It aims to evaluate the logical reasoning, deception detection, and multi-round dialogue understanding capabilities of large language models (LLMs) in complex social environments. It provides a standardized testing platform for studying the social reasoning capabilities of LLMs. This dataset is designed around a hidden role reasoning game, simulating social scenarios involving deception and misjudgment:

Role settings:

• Investigator: Always provide a truthful statement.
• Criminal: May selectively lie to confuse the public.
• Rumormonger: Thinks they are investigators, but their statements are randomly true or false.
• Lunatic: Believes himself to be a criminal and makes random true or false statements.

Game flow:

Each game consists of 3 rounds of dialogue, and in each round all players publicly identify whether a certain player is the criminal. Player 1 (i.e. the model perspective) needs to infer the real criminal and his own true role (possibly one of the four above) based on the three rounds of dialogue records. The core challenge is to distinguish true statements from random lies, deal with character self-perception biases (such as the false identity of rumor spreaders and lunatics), and gradually eliminate impossible options and lock in the only solution through logical contradictions or consistencies in multiple rounds of dialogue.

Citation

If you use this dataset or the SocialMaze benchmark in your research, please cite our work: bibtex @article{xu2025socialmaze, title={SocialMaze: A Benchmark for Evaluating Social Reasoning in Large Language Models}, author={Xu, Zixiang and Wang, Yanbo and Huang, Yue and Ye, Jiayi and Zhuang, Haomin and Song, Zirui and Gao, Lang and Wang, Chenxi and Chen, Zhaorun and Zhou, Yujun and Li, Sixian and Pan, Wang and Zhao, Yue and Zhao, Jieyu and Zhang, Xiangliang and Chen, Xiuying} year={2025}, note={Under review} }