One-sentence Summary

Washington University in St. Louis researchers evaluate risky decision-making across 20 frontier and open LLMs alongside a matched human subjects experiment, finding that reasoning models behave rationally while conversational models exhibit human-like biases and a large description-history gap, with training for mathematical reasoning identified as the key differentiating factor.

Key Contributions

• A controlled two-option risky-choice benchmark evaluates 20 frontier and open LLMs against matched human choices and an expected-payoff-maximizing rational-agent reference. It varies prospect representation as explicit descriptions or outcome histories, explanation prompting, option order, and gain/loss framing.

• LLM risky-choice behavior separates into reasoning models and conversational models, where reasoning models approximate expected-payoff maximization and are relatively insensitive to prospect order, gain/loss framing, explanation, and representation, while conversational models are less rational, more context-sensitive, and sometimes closer to human deviations.

• All evaluated models exhibit a description-history gap, larger for conversational models than reasoning models. Explanation prompting shifts choices, often making conversational models more aligned with the rational agent under brief explanations, while paired comparisons of open LLMs indicate that fine-tuning for mathematical reasoning differentiates the two clusters.

Introduction

Large language models are increasingly used in decision support and agentic workflows, making their behavior under uncertainty relevant for economic and practical applications. Traditional decision models assume agents maximize expected payoff, but LLMs may not follow this pattern because their competence emerges from broad training methods such as next-word prediction, instruction finetuning, human preference alignment, and mathematical reasoning.

Prior work on LLM decision-making has produced conflicting findings about rationality and alignment with human behavioral models. Much of this work also focuses on a single theory, uses limited model coverage, ignores how risky options are represented, or lacks direct comparisons with human choices under matched conditions.

The authors address these gaps by studying 20 frontier and open LLMs alongside human subjects in a controlled two-option risky-choice benchmark. They compare explicit prospect descriptions with outcome-history representations and test whether asking for explanations changes choices. Their work provides a clearer account of how model family, task representation, and explanation prompting shape LLM economic behavior.

Dataset

• The dataset is a structured evaluation set for comparing LLM and human choices among prospects. The core stimuli are three base prospect pairs. These pairs are drawn to span different outcome scales and are intentionally similar to, but not duplicates of, classic behavioral-economics tasks, which helps reduce memorization effects in LLMs.

• The prospect pairs are represented in two complementary forms:

  • Explicit prospects: Each option is shown directly as payoff-probability pairs, for example a 70 percent chance of $100 and a 30 percent chance of$100anda30percentchanceof0.
  • Implicit prospects: The same underlying distributions are converted into simulated histories, which are sequences of realized payoffs from repeated selection. The histories use 20 and 100 payoff samples.

• The presentation of histories is treated as a data subset condition. Results are aggregated across the 20-trial and 100-trial history sizes. The dataset therefore includes the same decision problems under both explicit description and sampled history presentation.

• Each decision instance also includes a prompt-style condition:

  • No explanation requested: the model outputs only the choice.
  • Short explanation requested: the model gives a one-sentence justification.
  • Mathematical or reasoning-style explanation requested: the model gives a brief quantitative or reasoning-based justification.

• A separate human-subject dataset is collected to benchmark model behavior. It contains responses from 360 U.S.-based participants recruited through Prolific. Participants make choices among the same prospect pairs used for the models and are compensated at an average rate of $24 per hour.

• The human data are treated as one population distribution over choices. The study received institutional IRB approval. An initial attention filter excluded participants who averaged less than 8 seconds per question, but the main results include all participants because the exclusion did not materially change outcomes.

• In use, the dataset is not described as a training corpus. It is used as an evaluation and comparison resource for model and human decisions. The main processing steps are converting explicit distributions into simulated histories, aggregating over history sample sizes, varying explanation prompts, and pairing model choices with human choice distributions.

Method

The method is organized as a controlled behavioral evaluation framework for measuring decision-making patterns in large language models. Rather than treating model outputs as binary answers, the authors treat each decision context as a sample from a response distribution and then analyze that distribution with interpretable models of choice behavior.

The first component of the framework is a structured testbed of choice prospects. Each trial presents a pair of alternatives with defined payoff and probability characteristics. These prospects are used to test how models respond when information is presented in different formats. The authors deliberately draw inspiration from classic behavioral decision settings while constructing novel stimuli to reduce the possibility of memorization.

A central design element is the distinction between explicit and implicit presentation of prospects. In the explicit condition, each prospect is stated directly as a set of payoff-probability pairs, such as a specified chance of receiving a gain and a specified chance of receiving zero. In the implicit condition, the same underlying distribution is represented through a simulated history, meaning a sequence of sampled outcomes that a decision-maker would observe after repeatedly choosing that prospect. The authors compare behavior across these presentation modes to capture a description-history gap, where identical underlying prospects can induce different choices depending on whether the evidence is described symbolically or experienced sequentially. Simulated histories are evaluated at multiple lengths, allowing the method to test whether repeated exposure to a payoff pattern changes model behavior in a measurable way.

A second design element is the inclusion of explanation prompts. For each choice, the framework supports multiple response styles: outputting only the selected prospect, providing a brief one-sentence justification, or providing a short mathematical or reasoning-style explanation. This allows the pipeline to assess whether asking for a rationale changes the model’s choice distribution, not merely its generated text.

To interpret observed choices, the authors fit parametric models based on prospect theory. These models translate contextual choices into risk preferences and probability weighting patterns. One specification uses parameters for risk preference, loss aversion, probability weighting, and decisiveness. However, because loss aversion and decisiveness can be difficult to separate in pure-loss contexts, the authors use an alternative parameterization as the primary model. This specification retains risk preference and probability weighting parameters while replacing the loss aversion and general decisiveness parameters with separate decisiveness parameters for gains and losses. The resulting interpretation is as follows: larger risk preference parameters indicate risk seeking, probability weighting parameters indicate whether small probabilities are overweighted or underweighted, and frame-specific decisiveness parameters indicate how strongly utility differences influence choices in gain or loss domains. The ratio of loss-domain decisiveness to gain-domain decisiveness serves as a proxy for loss aversion. To avoid estimation pathologies, all parameters are constrained to a finite range, with highly deterministic expected utility behavior corresponding to large decisiveness values and neutral risk and probability treatment.

Behavior is quantified by repeatedly querying each model under fixed conditions. Each context is evaluated multiple times using identical inputs, and the resulting valid outputs are converted into empirical choice rates. Invalid outputs are excluded from rate calculations but retained as raw samples. The authors also note that increasing the number of sampled responses beyond a moderate number produces minimal qualitative change while increasing cost, so the pipeline uses a practical number of repeated queries to estimate a stable response distribution.

Let $x$x denote a full decision context, including the pair of prospects, presentation format, prospect order, gain or loss framing, and explanation style. Let $p_m(x)$pm​(x) denote the probability that model $m$m chooses a reference prospect in context $x$x. This probability is estimated empirically for language models and human subjects, and it can also be computed from a fitted parametric model. Model similarity and goodness of fit are measured across many such contexts. One measure is the mean squared error between the response distributions of two models:

This measures absolute misfit, but it does not fully capture whether models respond coherently across contexts. To capture this, the authors also treat response probabilities as distributions over contexts and compute the Pearson correlation between the response vectors of two models:

Correlation is used as the primary similarity and goodness-of-fit measure because it reflects whether changes in one model’s choice tendency across contexts are matched by similar changes in another model’s behavior. When evaluating fitted parametric models, these measures are computed on held-out data to assess generalization rather than in-sample fit.

The framework also measures behavioral stability. Decisiveness captures how confidently a model selects one prospect over the other. For a model $m$m, this is defined as

Higher values indicate more deterministic choices, while values closer to chance indicate greater uncertainty or randomness.

Consistency is measured in three forms: order consistency, prompt consistency, and frame consistency. Order consistency tests whether switching the order of presented prospects changes the choice probability. Prompt consistency tests whether the type of explanation requested changes the choice probability. Frame consistency tests whether gain versus loss framing changes behavior for otherwise equivalent prospects. For order and prompt consistency, let $S(x)$S(x) denote the set of controlled variations of context $x$x. The consistency measure is then

A value near one indicates that behavior remains stable across the varied prompt conditions, while lower values indicate sensitivity to superficial contextual changes. Frame consistency is defined by comparing gain-framed and loss-framed versions of the same decision:

Together, these modules form a pipeline that converts raw model outputs into stable decision distributions, fits interpretable behavioral models to those distributions, and evaluates models not only by their average error but also by whether they exhibit coherent, decisive, and frame-stable choice behavior.

Experiment

The evaluation compares choices made by 20 language models under uncertainty with a human sample and a risk-neutral expected-payoff maximizer, using both explicit and implicit prospect presentation. The experiments establish a behavioral split between reasoning models and conversational models: reasoning models are more consistent, decisive, and close to rational expected-payoff maximization, whereas conversational models are more variable and sensitive to order, framing, and prompt details. Additional experiments and model fits validate that implicit histories shift many models toward more human-like and less rational behavior, especially for conversational and smaller models, while reasoning training and larger scale improve predictability and alignment with economicus.

The fitted dual-beta parameters suggest that frontier reasoning models make explicit-prospect decisions that are close to rational and very predictable, with values near one and very high gain and loss decisiveness. Chat models generally show similar probability weighting but lower decisiveness and weaker model fit, with some models being much less consistent in loss decisions. Frontier reasoning models exhibit very high decisiveness and near-perfect correlation with the simple prospect theory model. Chat models are less decisive and less predictable than reasoning models, and one model shows an extreme loss-decisiveness value.

For implicit prospects, reasoning models display parameters close to the rational benchmark, with value and probability weighting near neutral and very high decisiveness in both gain and loss domains. These models are also highly predictable by the dual-beta prospect theory model, showing strong correlations and low fitting errors. Chat models are less deterministic and less well fit, often having much lower decisiveness, evidence of probability overweighting, and weaker predictive correlations with higher errors. Reasoning models remain close to rational behavior under implicit prospects. Chat models show much lower decisiveness and weaker predictability than reasoning models. Probability overweighting appears among several chat models, while one chat model remains near neutral in probability weighting.

The table shows that reasoning-trained models generally exhibit stronger and more consistent decision parameters than instruction-tuned variants. Reasoning variants have markedly higher decisiveness values and much lower error, indicating more predictable fit to the model. Reasoning-trained variants have substantially higher gain and loss decisiveness than comparable instruction-tuned models. Several instruction-tuned models show near-zero loss decisiveness alongside much higher gain decisiveness. Reasoning variants achieve much lower MSE and near-perfect correlation than instruction-tuned counterparts.

Reasoning-trained open models tend to fit implicit-prospect behavior better than instruction-tuned counterparts, especially through higher correlation and lower error. Their larger decisiveness parameters separate thinking variants from conversational variants, though the effect is not as uniform as in explicit prospect settings. Reasoning variants generally show much higher decision decisiveness than instruction-tuned variants. The correlation with model predictions is stronger for thinking variants, indicating more predictable behavior.

The experiments use dual-beta prospect theory fits to evaluate whether model decisions under explicit and implicit prospects are rational, decisive, and predictable, comparing frontier reasoning models with chat models and reasoning-trained variants with instruction-tuned variants. Frontier reasoning models are close to rational, highly decisive, and strongly predicted by prospect theory, while chat models are less decisive, less predictable, and sometimes show probability overweighting or weak consistency. The comparisons further show that reasoning-trained variants behave more coherently than instruction-tuned counterparts, with the effect strongest in explicit prospect settings and still present but less uniform under implicit prospects.