One-sentence Summary

Researchers from Carnegie Mellon University, KAIST, LG AI Research, and others introduce BIGGEN BENCH, a principled generation benchmark that evaluates nine distinct LM capabilities across 77 diverse tasks using instance-specific criteria to mirror human-level granularity, applied to 103 frontier LMs via five evaluator LMs.

Key Contributions

• BIGGEN BENCH is introduced as a generation benchmark for nine core language-model capabilities across 77 diverse tasks. It uses a principled top-down construction process to assign evaluation criteria tailored to each instance.

• The benchmark extends LM-as-a-judge evaluation beyond instruction following by applying evaluator language models to arbitrary, nuanced criteria across capabilities including reasoning, safety, tool usage, and multilingual abilities. This avoids reliance on exact-match or similarity metrics for assessing free-form outputs.

• The paper evaluates 103 frontier LMs using five evaluator LMs and examines performance trends among pre-trained, post-trained, and proprietary models. It also shows significant correlation between evaluator-LM judgments and human evaluations, while releasing code, data, and results.

Introduction

The paper is about systematically measuring what language models can actually do, especially for free-form outputs where a model may sound fluent but still fail a specific capability. This matters because better evaluation helps reveal model limitations, guide post-training, and compare frontier systems beyond simple helpfulness scores or multiple-choice accuracy.

Prior methods are limited because broad proxies like “helpfulness” are subjective, classification or similarity scores can miss false negatives, and many generation benchmarks focus on narrow capabilities or easily verifiable constraints. They also struggle to evaluate nuanced instance-level behaviors such as reasoning steps, tool use, theory of mind, cultural sensitivity, or response refinement.

The authors address this by introducing BigGen Bench, a top-down generation benchmark with instance-specific evaluation criteria across nine core capabilities, including instruction following, grounding, planning, reasoning, refinement, safety, theory of mind, tool usage, and multilingualism. They leverage evaluator LMs within a unified pipeline to score free-form outputs, evaluate 103 frontier and 14 proprietary models, and validate evaluator reliability through significant correlation with human judgments.

Dataset

• The authors introduce BiGGEN Bench as an evaluation benchmark for generation tasks, organized around a capability, task, instance, and evaluation criteria hierarchy. The final dataset contains 9 capabilities, 77 tasks, and 765 instances.

• Handcrafted seed subset: The initial data are manually annotated. Eighteen coauthors each contribute to one capability, creating instances across five tasks, while ten native speakers of Korean, Kazakh, Bengali, Spanish, Indonesian, German, French, Arabic, Russian, and Thai contribute multilingual instances. After reviewing grammar, fluency, and task difficulty, the authors remove 23 tasks and retain 77 tasks with 385 instances.

• Augmented subset: The authors expand the seed data using GPT-4-0125, guided by human-crafted examples as in-context demonstrations rather than zero-shot generation. For each new instance, five candidate outputs are generated, the candidate with the lowest BertScore semantic similarity is selected, and the selection step is repeated five times. Quality validation revises or discards weak examples, increasing the dataset to 770 instances across 77 tasks.

• Cross-validated final subset: Each coauthor reviews two capabilities, ten tasks, and fifty instances that they did not annotate. Instances flagged by one annotator are revised, while instances confirmed by two annotators as misaligned or having poor references or rubrics are removed. This filtering removes five instances and yields the final 765 instances.

• Metadata and schema: Each instance is linked to a capability, a task, and instance-specific evaluation criteria, including references or rubrics used to score generated responses. The paper describes no visual cropping strategy, as the benchmark is text-generation oriented.

• Use in the paper: The dataset is used for benchmark evaluation rather than model training. No training split or mixture ratio is reported. Evaluator language models score model responses using the benchmark’s criteria, and human judgments are collected for selected responses, including four LMs and six LMs for the multilingual capability, to verify evaluator reliability.

Method

The BIGGEN BENCH construction follows a controlled top down pipeline organized around a capability, task, instance, and evaluation criteria hierarchy. This structure keeps each prompt and expected output tied to a targeted generative capability, while instance specific references and score rubrics enable fine grained evaluation.

The first stage relies on expert authored seed data. The authors leverage eighteen coauthors to create task specific instances across multiple capabilities, with additional native speakers supporting multilingual tasks. Tasks are first drafted, then refined through collaborative discussion, and finally reviewed for grammar, fluency, and challenge level. Tasks that contain confusing or insufficiently difficult instances are removed. This produces a curated set of human written seed instances that serve as the quality baseline for later expansion.

The second stage expands the seed set through LLM assisted augmentation. The authors use GPT-4-0125 to generate additional instances, but they do not generate new prompts in a zero shot manner. Instead, human authored examples from each task are used as in-context demonstrations, preserving task format, expected reasoning style, and evaluation semantics. To promote diversity, the pipeline samples five candidate instances for each new prompt and selects the candidate with the lowest semantic similarity to existing instances, using $BertScore$BertScore as the similarity measure. This candidate filtering step is repeated, resulting in a larger pool of semantically varied but task aligned instances.

The third stage applies human cross validation to the augmented pool. Evaluators review instances that they did not originally author and assess whether the instance fits its assigned task, how difficult it is, whether the reference answer is acceptable, and whether the score rubric is usable. The cross validation results are summarized as follows:

The authors adopt a conservative filtering policy. If both evaluators judge an instance as bad, it is removed. If only one evaluator flags an issue, the instance is iteratively revised rather than discarded. This process removes misaligned prompts and defective references or rubrics, improving reliability before final benchmark release.

The final stage collects human judgments on model responses to validate the evaluation protocol. Human annotators grade responses produced by selected large language models under the benchmark rubrics. The human evaluation uses a staged workflow with recruitment, qualification, and main evaluation phases to improve rater consistency. These human scores provide a calibration signal for assessing whether automated evaluator LMs produce accurate and stable model comparisons.

Experiment

The experiments evaluate BIGGEN BENCH across 103 language models using direct LM-as-judge scoring, and they examine how performance scales with model size, training stage, and model type. Base models show smoother scaling benefits, while chat models depend more on post-training factors beyond parameter count, with the base-chat performance gap narrowing as models become larger. Additional validation experiments show that evaluator language models, especially when combined through majority voting, correlate reasonably with human judgments, and that instance-specific fine-grained rubrics improve evaluation accuracy while largely avoiding verbosity bias.

Larger base language models tend to score higher across capabilities on a logarithmic scale, with a relatively strong relationship for average performance. Chat models also show positive scaling, but the association with parameter count is weaker, suggesting other factors affect post-trained behavior. The capability-level trends are mostly positive across grounding, instruction following, planning, reasoning, refinement, safety, theory of mind, and tool use. Base models show a clearer size-performance relationship than chat models. Chat model scores increase with scale, but less predictably than base model scores. Theory of mind and tool use are included in the scaling analysis, and the surrounding evaluation notes indicate lower human agreement for these capabilities.

All listed capability coefficients are statistically significant and negative, showing that the interaction between chat versus base model group and parameter size systematically affects performance. This trend is consistent with the finding that larger models reduce, but do not necessarily eliminate, the gap between base and chat LMs. Darker shading indicates that some capability gaps can remain wide even as parameter size increases. Refinement has the least negative interaction among the listed capabilities, while safety has the most negative. Parameter scaling therefore narrows base to chat performance differences overall, with strength varying by capability.

Evaluator language models show measurable agreement with human judgments across multiple capabilities, with GPT-4-Turbo and a majority-vote ensemble achieving the strongest overall alignment. Prometheus-2-BGB 8x7B performs close to proprietary evaluators, and self-consistency decoding can improve agreement. Theory of mind and tool-use correlations remain relatively weaker than other capability areas. Open-source Prometheus-2-BGB is competitive with proprietary evaluator LMs in agreement with human ratings. Self-consistency decoding raises agreement, particularly for the Prometheus-2 family. Theory of mind and tool-use judgments correlate less strongly with human ratings than other capabilities.

Proprietary language models generally show larger capability advantages over open-source models, with the average gap marked by a moderate effect size. The gap is not uniform across capabilities, appearing more pronounced for tool usage and planning, while smaller for safety and instruction following. This suggests that open-source models are relatively closer to proprietary models on some core capabilities than on others. Proprietary models generally outperform open-source models across the measured capabilities. The advantage is especially strong for tool usage and planning. Safety and instruction following show comparatively smaller gaps between proprietary and open-source models.

The experiments examine capability scaling across base and chat models, alignment of evaluator language models with human judgments, and performance gaps between proprietary and open-source systems. Larger models generally improve across capabilities, with base models showing a clearer scaling trend than chat models, while parameter growth narrows but does not fully remove base-chat differences, especially for capabilities such as safety and refinement. Evaluator models agree well with human ratings, particularly strong proprietary evaluators, ensemble approaches, and open-source Prometheus-2 variants, although theory of mind and tool-use judgments remain less consistent; proprietary models also outperform open-source models overall, with larger gaps in tool use and planning than in safety and instruction following.