One-sentence Summary

University of Washington, CMU, Allen Institute for AI, et al. introduce INFINITY-CHAT, a 26K-query dataset of open-ended prompts with a six-category taxonomy, revealing an Artificial Hivemind effect in which LMs exhibit pronounced intra-model repetition and inter-model homogeneity that pose long-term AI safety risks.

Key Contributions

• INFINITY-CHAT introduces a dataset of 26K real-world, open-ended user queries and a taxonomy with 6 top-level categories and 17 subcategories for prompts that admit many plausible answers and lack a single ground truth.

• INFINITY-CHAT enables a large-scale study of mode collapse in language models, identifying an Artificial Hivemind effect characterized by intra-model repetition and inter-model homogeneity in open-ended generation.

• The resource contributes 31,250 human annotations across absolute ratings and pairwise preferences, with 25 independent annotations per example, and shows that state-of-the-art LMs, reward models, and LM judges are less calibrated to idiosyncratic human preferences even when overall quality is comparable.

Introduction

Large language models are central to modern AI assistants, but they often lack the diversity needed for open-ended interactions where many answers may be reasonable. This matters because repeated similar outputs can narrow human expression and undermine pluralistic alignment. Earlier work has mainly measured diversity with stylized or narrow tasks, and it has lacked scalable benchmarks built from naturally occurring user requests that expose pluralistic preferences. The authors introduce INFINITY-CHAT, a dataset of 26K real-world open-ended queries mined from WildChat, along with a taxonomy of six top-level categories and 17 subcategories. Using this dataset, they show an Artificial Hivemind effect across many open and closed LMs, where a single model repeats itself and different models converge on similar ideas despite minor phrasing differences. They also collect dense human annotations on 31,250 response evaluations and find that LMs, reward models, and LM-based judges are often miscalibrated when responses elicit divergent but comparable-quality preferences. Together, the work provides a benchmark and framework for evaluating, diagnosing, and mitigating mode collapse and pluralistic misalignment in modern LMs.

Dataset

• The authors build INFINITY-CHAT from real-world, single-turn GPT-4 user inputs in WildChat. The starting pool has 37,426 high-quality English queries that are non-toxic and 15 to 200 characters long.

• GPT-4o filters and classifies each query for whether it seeks meaningful information, is a greeting or model inquiry, and allows a single valid response or multiple valid responses. Ambiguous queries are revised for clarity.

• The final INFINITY-CHAT query set contains 26,070 open-ended queries and 8,817 closed-ended queries. The paper uses these as real-world prompts for studying diverse and pluralistic language model responses.

• The authors create a taxonomy for open-ended queries. They begin with about 100 mined queries, manually assign tentative labels, and refine them into 6 high-level categories and 17 fine-grained subcategories.

• GPT-4o then labels the full open-ended query set, assigning one or more existing categories and detecting novel categories beyond the seed set. This produces 314 novel categories.

• The category distribution includes Creative Content Generation at 58.0%, Alternative Writing Genres at 38.5%, Concept Explanation at 23.6%, Skill Development at 23.5%, Analytical and Interpretive Questions at 22.6%, Hypothetical Scenarios at 22.2%, and Brainstorming and Ideation at 15.2%.

• The paper does not describe a standard training split, validation split, or data mixture ratios for INFINITY-CHAT in the provided text. The dataset is mainly used as a prompt and annotation resource for evaluating response diversity and human preference distributions.

• For dense human preference annotation, the authors use INFINITY-CHAT100 and sample 50 prompts from it. For each prompt, they collect 15 alternative responses and gather 25 absolute quality ratings per response on a 1 to 5 scale.

• The absolute rating subset contains 25 × 15 × 50 = 18,750 Query, Response labels. These labels are used to analyze variation in human judgments of response quality.

• For pairwise preference annotation, the authors sample 10 response pairs per prompt and collect 25 annotations per response pair. The annotations record strong or weak preferences between two responses to the same query.

• The pairwise subset contains 25 × 10 × 50 = 12,500 Query, Response Pair labels. These dense preference labels support analysis of disagreement and entropy in human judgments.

• The dataset metadata includes query text, query category, response candidates, absolute quality scores, and pairwise preference distributions. No cropping strategy is described, because the data consist of text queries and annotated response preferences rather than image crops.

Method

The authors leverage a data-centric pipeline to construct a query inventory and a preference annotation workflow that captures diversity in open-ended prompts. The first component starts from in-the-wild user-chatbot logs and applies conservative quality filters, retaining English, non-toxic, single-turn interactions of usable length. A large language model then screens each prompt for meaningful information seeking and for whether the request admits one valid answer or multiple valid answers. Ambiguous prompts are rewritten for clarity before entering the final pool.

The method then builds a taxonomy of open-ended queries through a semi-automatic cycle. A small seed set of mined queries receives human tentative labels, and these labels are iteratively merged, split, and grouped into a hierarchical structure. Once the seed taxonomy is stable, the authors scale annotation with GPT-4o, which assigns one or more categories to each user query and flags novel types that are not covered by the original labels. The resulting taxonomy organizes diverse request styles, from creative generation and brainstorming to concept explanation and analytical questions. Refer to the framework diagram below:

The diagram shows the six high-level groups, their seventeen subcategories, and the relative frequency of each query type. Beyond the fixed taxonomy, the labeling stage identifies additional recurring themes. These novel categories are summarized through a word cloud that highlights terms such as cultural, ethical, historical, media, and humor related dimensions. The discovered dimensions are visualized below:

For preference analysis, the authors design a dense annotation workflow instead of relying on sparse labels. They select a representative subset of open-ended prompts and generate multiple alternative responses for each prompt. Two annotation schemas are collected. Absolute quality annotations are gathered on a 1 to 5 scale, with 15 responses sampled for each of 50 prompts and 25 human ratings collected per query-response pair, yielding $25 \times 15 \times 50 = 18{,}750$25×15×50=18,750 absolute labels. Pairwise annotations compare two responses to the same query using strong or weak preference labels plus neutral options. For this setting, 10 response pairs are sampled per prompt, and 25 annotations are collected for each query-response-response triplet, totaling $25 \times 10 \times 50 = 12{,}500$25×10×50=12,500 pairwise labels.

The collected annotations are then converted into distributional signals by computing Shannon entropy over the label distribution for each annotated item. For a pairwise triplet with empirical category probabilities $p_k$pk​, the entropy is given by

High entropy indicates substantial disagreement among annotators, while low entropy suggests consensus about which alternative response is better or whether the responses are similarly acceptable. The entropy distribution and representative examples are shown below:

The left panel summarizes the overall disagreement trend, while the right panels illustrate cases in which many response pairs receive near-uniform preference support. A parallel entropy computation is applied to absolute ratings, treating the five score levels as a distribution. This reveals that even scalar quality judgments are often heterogeneous for open-ended prompts. The corresponding entropy analysis is presented below:

The middle and right panels show that query-response pairs can have very different label spreads, with some responses receiving concentrated high ratings and others receiving dispersed scores across all quality levels.

Experiment

The experiments examine whether large language models produce diverse outputs to open-ended queries and whether model-generated ratings reflect human judgments of such outputs. By sampling many responses from numerous models on real-world open-ended questions, the work finds substantial repetition within individual models and notable homogeneity across different model families, even under diversity-promoting decoding. The evaluation experiments further show that language models, reward models, and LM judges align less well with human scores when alternative responses are of comparable quality or when human annotators disagree, suggesting current systems underestimate the validity of multiple reasonable answers.

The analysis shows that responses to the same open-ended query are highly repetitive, with most models producing many queries whose average pairwise response similarity exceeds 0.8. Randomly paired responses from different queries fall far lower, in the 0.1 to 0.2 similarity range. This suggests that sampling parameters do not prevent substantial semantic repetition within response pools. Same-query responses usually fall in very high embedding similarity ranges. Most models show dominant repetition in similarity bands above 0.8. Cross-query random pairs are much less similar than same-query response pairs.

Under the reported min-p sampling setting, responses generated by the same language model to the same open-ended prompt tend to be very similar. Across models, most pairwise similarity scores fall in the high ranges, indicating substantial repetition rather than diverse completions. Most same-prompt response pairs are more similar than different. High pairwise similarity is common across the tested language models. Model families differ in how strongly their outputs cluster in the highest similarity ranges.

Under the reported sampling settings, average pairwise sentence embedding similarities show substantial semantic overlap among responses from different language models. This overlap is broad across model families, and qualitative examples indicate that outputs to fully open-ended queries can be strikingly similar, including extended verbatim spans. In related evaluation analyses, model judgments align less with human ratings when response pairs are very similar in quality or involve high annotator disagreement. Different language models show high pairwise embedding similarity on open-ended generation tasks. Model evaluations are less aligned with human ratings when responses are similar or annotator disagreement is high.

The experiments evaluate embedding similarity among model responses to open-ended queries under the reported sampling settings. They show that responses to the same prompt, especially from the same model, are highly repetitive and often semantically near identical, while unrelated query pairs are much less similar. The analysis further indicates that different models can produce strikingly similar outputs, and that model evaluations track human judgments less closely when response pairs are very similar or annotator disagreement is high.