Typed Decisions is a typed probabilistic decision benchmark dataset released by LocalLLaMA, designed to evaluate a model’s ability to answer multiple typed questions simultaneously based on unstructured state information and output corresponding probability distributions, with a focus on decision accuracy and probabilistic calibration performance.

The dataset is built around four business scenarios: agent trace observability, customer service, invoice processing, and security incident response. It contains 1,600 synthetic cases in total, including 1,200 training cases and 400 test cases. Each case consists of one state information and five typed questions, and the test set involves 2,000 decisions in total. The dataset uses three decision types: noul, choice, and score, supporting true/false probability judgment, multi-option probability distributions, and ordinal score distributions, respectively. It also provides gold-standard probability distributions and related auxiliary fields, which can be used to evaluate a model’s structured decision-making and probabilistic prediction capabilities across different business scenarios.

Dataset Composition

• Workflow types: include four business scenarios: agent trace observability (agent_trace_observability), customer service (customer_service), invoice processing (invoice_processing), and security incident response (security_incidents);
• Input fields: state and questions together constitute the model input, and the textual judgment criteria for options are stored in the criteria field;
• Gold-standard fields: gold stores the complete gold-standard distribution; __label, __probabilities, __score, and __probability_true provide the corresponding flattened results;
• Auxiliary fields: factors and label_agreement record the case generation factors and annotation agreement, respectively, and are not used as model input;
• Decision types: include noul, choice, and score, used respectively to predict the probability that a statement is true, the probability distribution over multiple options, and the probability distribution over ordered score levels and the expected score.