One-sentence Summary

NVIDIA researchers introduce HELPSTEER, a $37\text{k}$37k-sample multi-attribute helpfulness dataset annotating correctness, coherence, complexity, and verbosity alongside overall helpfulness to address artifact learning in existing preference data, and training Llama 2 $70\text{B}$70B via STEERLM yields a $7.54$7.54 MT Bench score—the highest for open models without relying on GPT-4-generated training data—with the dataset released under $\text{CC-BY-4.0}$CC-BY-4.0.

Key Contributions

• Introduces HELPSTEER, a multi-attribute helpfulness dataset with 37,000 samples, where each response is annotated for correctness, coherence, complexity, and verbosity in addition to overall helpfulness, offering finer-grained preference signals than existing ranking-only datasets.
• Training Llama 2 70B on HELPSTEER with the STEERLM technique achieves a score of 7.54 on MT Bench, the highest reported among open models that do not require training data from proprietary models such as GPT-4.
• Publicly releases the dataset under a CC-BY-4.0 license, enabling the community to build upon and replicate the work.

Introduction

Aligning domain-general language models to follow user instructions typically centers on the twin goals of helpfulness and safety, yet the notion of helpfulness itself remains poorly defined. Recent approaches sidestep this by relying on blackbox user or annotator preferences, but that strategy is computationally and data intensive, intellectually unsatisfying, and risks teaching models to equate length with quality. Prior attempts to define helpfulness used attributes like creativity and humor, which only suit specific contexts such as story writing and can backfire in formal or task-oriented settings. Domain-specific studies show helpfulness factors vary by application, for example summarization values accuracy, coverage, and coherence, while long-form question answering emphasizes relevance, factuality, and completeness.

The authors propose that helpfulness in domain-general responses can be assessed along four dimensions: correctness, coherence, complexity, and verbosity. Correctness covers the inclusion of all pertinent facts without errors, coherence captures consistency and clarity of expression, complexity reflects the intellectual depth of a response, and verbosity relates to the amount of detail, which correlates positively with human preference. To validate this framework, they curate HELPSTEER, a dataset of 37,000 conversations where each response is annotated for these four attributes plus overall helpfulness. They also highlight limitations of existing data sources, noting that many datasets use GPT-4 annotations without human validation, raising concerns about biases and potential legal conflicts under OpenAI's Terms of Use, which restrict using outputs to develop competing models.

The authors train a Llama 2 70B model on HELPSTEER using the STEERLM technique, achieving the highest MT Bench score of 7.54 among models that do not rely on training data from powerful proprietary systems. The resulting model is more truthful and coherent than Llama 2 70B Chat and other baselines while remaining user steerable in verbosity and complexity. The dataset is openly released under a CC-BY-4.0 license to support further research and community building.

Dataset

The authors introduce HELPSTEER, a dataset designed to address limitations encountered when applying the STEERLM technique to the Open Assistant dataset. Original responses showed issues such as factual inaccuracy, incoherence, oversimplification, or excessive verbosity, especially on tasks that involve a reference text, including Rewrite, Summarization, Classification, Extraction, and Closed Question Answering.

Dataset composition and sources:

• The dataset contains 37,120 high-quality annotated samples built from 10,459 single-turn prompts
• Roughly half of the prompts were created by Scale AI, an external data annotation vendor, and the other half were synthetically generated using templates to diversify prompt sources
• A larger initial set of prompts was produced, from which about 20% were filtered out as unsatisfactory

Prompt distribution:

• Categories include Open Question Answering, Generation, and Brainstorming, plus the five reference-text tasks mentioned above
• Each category holds approximately 10% of prompts, with the five underperforming tasks sharing the remaining 20% to improve their representation

Response generation:

• Four distinct responses per prompt were generated using an in-house 43 billion parameter model
• Generation settings: maximum context length of 4,096 tokens, temperature 1.0, top_p 0.80, top_k 1000, and a repetition penalty of 1.0, producing diverse but reasonable outputs

Annotation process:

• Each response was rated independently on five attributes: Helpfulness, Correctness, Coherence, Complexity, and Verbosity, using a Likert-5 scale from 0 to 4
• Unlike RLHF annotations that compare responses to the same prompt, independent rating scales annotation workload linearly instead of quadratically
• About 200 U.S.-based annotators via Scale AI were selected after English proficiency screening, an introductory training course, and a test involving 35 sample responses
• Each annotation received at least two human reviews plus automated checks, followed by the authors' own quality assurance filtering

Data statistics and analysis:

• HELPSTEER contains 37.1k conversations, comparable to Open Assistant's 59.4k
• Prompts average 2,491.8 characters (std 1,701.7), substantially longer than Open Assistant's 397.5 characters (std 620.8), mainly due to reference texts in Summarization, Rewrite, and Extraction tasks
• Responses average 497.3 characters (std 426.7)
• Average attribute ratings: coherence 3.30, correctness 2.84, complexity 1.44, verbosity 1.53, and helpfulness 2.78 out of 4
• Correctness and coherence strongly correlate with helpfulness (Pearson R > 0.6), while complexity and verbosity correlate weakly (Pearson R > 0.2)
• An OLS regression with helpfulness as the dependent variable shows the four attributes account for 73.0% of the variance in helpfulness

Usage in the paper:

• The dataset is used with the STEERLM technique to obtain better annotations for attributes that contribute to helpfulness, particularly for the five reference-text-heavy tasks where performance was suboptimal
• The dataset serves as a training and evaluation resource for steering model responses on helpfulness-related attributes

Method

To address the limitations of the Open Assistant dataset when applied with the STEERLM technique, the authors construct a dedicated dataset, HELPSTEER, designed to capture fine-grained attributes of response helpfulness. The collection pipeline consists of four stages: prompt collection, response generation, response annotation, and quality assurance.

Prompt Collection. The authors gather 10,459 single-turn prompts, with approximately half created by an external annotation vendor and the remainder generated syntactically via templates. To ensure diversity and coverage, a larger initial prompt set is produced, from which around 20% of unsatisfactory prompts are filtered out. Following prior work, the collection spans Open Question Answering, Generation, and Brainstorming tasks, alongside five tasks that were underrepresented in the Open Assistant dataset: Rewrite, Summarization, Classification, Extraction, and Closed Question Answering. Each category is allocated roughly 10% of the prompts, while the remaining 20% is reserved for the five tasks where the base STEERLM model showed weaker performance.

Response Generation. Four distinct responses are generated for each prompt using an in-house 43 billion parameter model, with a maximum context length of 4,096 tokens. The generation configuration uses a temperature of 1.0, top_p of 0.80, a repetition penalty of 1.0, and top_k of 1000, yielding responses that are diverse yet reasonable.

Response Annotation. Each response is independently rated on five attributes: Helpfulness, Correctness, Coherence, Complexity, and Verbosity, on a Likert-5 scale from 0 to 4. Unlike RLHF annotations, which compare responses to the same prompt and lead to quadratic growth in annotation workload relative to the number of responses per prompt, HELPSTEER rates each response independently. This design increases annotation effort linearly. Approximately 200 U.S.-based annotators are engaged through a vendor. Candidates first complete preliminary assessments, including English proficiency tests, then undergo an introductory training course that concludes with annotating 35 sample responses. After annotation, the vendor performs quality assurance with at least two human reviews per annotation in addition to automated checks, and the authors independently verify the results. After filtering at multiple stages, the final dataset contains 37,120 high-quality annotated samples.

STEERLM Training. The authors train Llama 2 13B and 70B models following the STEERLM approach, a model alignment method that serves as an alternative to RLHF. STEERLM comprises four key steps. First, an Attribute Prediction Model is trained to predict scores for multiple semantic attributes that capture dimensions of response helpfulness, such as correctness and coherence. Second, prompt-response pairs are annotated with these attributes using the trained prediction model. Third, Attribute Conditioned Supervised Fine-Tuning (AC-SFT) is performed by training a foundation model on the annotated data to generate responses conditioned on specified attribute values. Fourth, the AC-SFT model can be further improved by bootstrapping additional training data through sampling the model to obtain diverse, high-quality responses.

Several modifications streamline the original STEERLM pipeline. Only the Open Assistant (OASST) dataset is used for AC-SFT training rather than multiple datasets. The attribute labels from OASST are scaled to a 0-4 range to align with the HELPSTEER annotations. For the Attribute Prediction Model, the OASST and HELPSTEER datasets are combined to predict nine labels in total: the Quality, Humor, Toxicity, and Creativity labels from OASST, plus the five labels from HELPSTEER. The bootstrapping step of sampling the AC-SFT model and retraining on its generations is excluded, as initial exploration showed minimal gains. Finally, instead of a language model based attribute predictor, the authors employ a regression model, which performs better. This is implemented by taking the last hidden state from the Llama 2 foundation model and adding a regression head on top for each attribute.

Training Hyperparameters. Both the Attribute Prediction and Attribute-Conditioned SFT models are trained for 800 steps with a global batch size of 128 (close to 2 epochs), a constant learning rate of 5e-6, and the AdamW optimizer. At inference time, all attributes are set to 4 except creativity, humor, and toxicity, which are set to 0, unless steerability is the focus of the experiment.

Experiment

The experiments evaluate STEERLM trained with the HELPSTEER dataset against baselines including SFT, RLHF, DPO on open-source datasets, and Llama 2 70B Chat, using both automatic metrics (MT Bench for helpfulness, TruthfulQA for correctness, perplexity for coherence, FKGL for complexity, and response length for verbosity) and human rankings with Elo scores. Results show that STEERLM produces the most helpful, correct, and coherent responses, outperforming all baselines in both automatic and human evaluations, with particular gains in instruction-following categories. Ablation studies confirm that all five HELPSTEER attributes contribute to helpfulness, with correctness being especially critical, while also revealing that optimizing solely for helpfulness can reduce factual accuracy. A steerability demo further demonstrates that users can effectively control response complexity and verbosity at inference time to suit different audiences and scenarios.

The table compares three open-source helpfulness preference datasets, highlighting differences in annotated attributes and dataset scale. HELPSTEER provides multiple fine-grained attributes, Open Assistant includes three broad ones, and HH-RLHF offers only pairwise preferences. An ablation analysis indicates that HELPSTEER's attributes are more impactful for model helpfulness than Open Assistant's, and combining both attribute sets yields the best performance. HH-RLHF is the largest dataset but lacks attribute-level labels, relying only on preferred and rejected response pairs. HELPSTEER has fewer conversations but includes detailed attributes such as correctness, coherence, complexity, and verbosity. Ablation results show that removing correctness or helpfulness from HELPSTEER reduces helpfulness, with correctness being critical for factuality. HELPSTEER attributes contribute more to helpfulness than Open Assistant attributes, and using both sets together produces the most helpful model.

The table reports descriptive statistics and correlations for helpfulness-related attributes in two datasets, showing that correctness and coherence strongly correlate with helpfulness, while complexity and verbosity have weaker associations. HELPSTEER responses tend to be coherent and moderately correct but low in complexity and verbosity, and these attributes collectively explain a large portion of the variance in helpfulness. Correctness and coherence exhibit strong positive correlations with helpfulness, whereas complexity and verbosity show only weak correlations. HELPSTEER responses have high average coherence, moderate correctness, and low complexity and verbosity, together yielding moderately helpful responses. The four attributes combined account for 73% of the variance in helpfulness according to regression analysis. Open Assistant attributes such as creativity and humor show weak or negative correlations with quality, indicating they are not essential for helpfulness.

STEERLM outperforms baseline models across automatic metrics, achieving the highest MT Bench score while requiring less compute than a strong RLHF baseline. Its responses are more helpful, correct, coherent, and appropriately verbose compared to open-source trained models and Llama 2 Chat. The results indicate that conditioning on multiple attributes like correctness, coherence, complexity, and verbosity improves response quality. STEERLM achieves the highest helpfulness score on MT Bench, surpassing all baselines including those trained with RLHF or DPO on open-source datasets. Compared to the best RLHF baseline, STEERLM attains a higher MT Bench score while using only a fraction of the compute, and alternative models with similar compute trail further behind. STEERLM produces responses that are more truthful, coherent (lower perplexity), and appropriately detailed, balancing verbosity between the succinct open-source trained models and the more verbose Llama 2 Chat. Compared to open-source trained models and Llama 2 Chat, STEERLM shows notable gains in instruction-heavy categories like coding, math, reasoning, extraction, and roleplay.

In human pairwise evaluations, the STEERLM model achieved the highest Elo score and win rates against both Llama 2 Chat and an RLHF baseline. It outperformed these models despite requiring less compute, and its responses were favored for helpfulness and correctness in direct comparisons. STEERLM attained the top Elo rating of 1050 with win rates of 57.5% against Llama 2 Chat and 62.9% against the RLHF baseline. Llama 2 Chat scored lower with a 42.5% win rate against STEERLM and an Elo of 979, while the RLHF baseline trailed with 971. The evaluation indicates that STEERLM produces more helpful and correct responses than both alternatives, even though one baseline required roughly five times more compute.

Removing any of the five HELPSTEER attributes during training lowers MT Bench, indicating each contributes to helpfulness. Ablations for helpfulness and correctness show that optimizing for helpfulness alone can reduce factual correctness, while removing correctness training substantially hurts overall helpfulness. Using all attributes together yields the best MT Bench, outperforming models trained with either attribute set alone or a vanilla SFT model. Removing any HELPSTEER attribute reduces MT Bench, confirming each attribute helps improve helpfulness. Dropping helpfulness increases TruthfulQA MC2, suggesting that explicit optimization for helpfulness can come at the cost of correctness. Removing correctness causes the largest MT Bench drop and also raises response complexity (FKGL), indicating that factual training is important for both usefulness and clarity. Removing the full HELPSTEER set lowers MT Bench more than removing the Open Assistant set, showing HELPSTEER attributes contribute more to helpfulness. Using both attribute sets together achieves the highest MT Bench, while either set alone still outperforms a vanilla SFT model.

Across experiments, the authors evaluate HELPSTEER datasets and the STEERLM model against baselines like Open Assistant, HH-RLHF, and RLHF-trained systems. Fine-grained HELPSTEER attributes, especially correctness and coherence, prove more impactful for helpfulness than broader Open Assistant attributes, with these factors explaining 73% of helpfulness variance. STEERLM achieves the highest MT Bench and Elo scores while using significantly less compute, surpassing baselines in human pairwise evaluations and across instruction-heavy tasks. Ablation studies confirm that every HELPSTEER attribute contributes to helpfulness, with correctness being critical, though explicit helpfulness optimization can slightly reduce factual accuracy.