One-sentence Summary

University of Maryland, College Park researchers use atomic hypothesis decomposition to form granular sub-problems in NLI and defeasible NLI, revealing that LLMs struggle with logical consistency at the atomic level and proposing an inferential consistency metric that measures whether models make consistently correct or incorrect predictions about the same fact across different contexts.

Key Contributions

• Atomic decomposition is applied to hypotheses in traditional NLI and defeasible NLI to form granular subproblems that probe model consistency, clarify the structure of non-monotonic reasoning, and measure diversity of examples in benchmark datasets.

• Experiments on atomic NLI and defeasible NLI subproblems, including δ-SNLI, show that LLMs struggle with logical consistency when reasoning over individual atomic inferences, and that atomic labels in defeasible reasoning relate to full-problem labels more complexly than in traditional NLI.

• Critical atoms are identified as the atomic inferences that most contribute to a defeasible NLI label, and a framework based on question under discussion groups examples by shared critical atoms to measure inferential consistency across contexts.

Introduction

Atomic decomposition breaks text into atomic propositions, or granular facts explicitly supported by the original text, which is useful for fact checking, claim verification, multihop QA, and finer-grained evaluation of model reasoning. Prior work has used decomposition for claim checking and span-level NLI, but it has largely focused on premise decomposition or models trained to use atomic inference, and it has not examined atomic reasoning in defeasible NLI, where conclusions can change when new evidence is introduced. The authors leverage hypothesis decomposition in NLI and defeasible NLI to create atomic sub-problems, analyze the logical consistency of large language models, and introduce critical atoms and inferential consistency measures to probe how model predictions for key inferences vary across contexts.

Dataset

• The authors use δ-NLI, the defeasible NLI benchmark introduced by Rudinger et al. (2020), which asks whether an update sentence strengthens or weakens a hypothesis given a premise and hypothesis.

• δ-NLI is built from three commonsense reasoning sources: SNLI, ATOMIC, and SOCIAL-CHEM-101. For SNLI and ATOMIC, crowdworkers write update sentences for neutral premise-hypothesis pairs. For SOCIAL-CHEM-101, the source provides only the hypothesis part.

• The authors mainly use δ-SNLI, the SNLI-derived split of δ-NLI. This subset selects neutral premise-hypothesis pairs from SNLI and pairs each with a crowd-written update sentence. The resulting examples are structured as premise, hypothesis, and update.

• The original crowdsourcing rules ask workers to produce updates with a target strengthen or weaken label, but they are not told which part of the hypothesis to target. Workers also may not write updates that directly contradict premise information.

• The train, development, and test splits are separated at the premise-hypothesis level to prevent leakage between examples that share the same underlying pair.

• For δ-SNLI-TEST, the authors construct atomic decompositions of hypotheses. Each atom is intended to be a valid atomic sub-problem that should entail from the original hypothesis but should not entail from the premise.

• The atomic decomposition pipeline has two filtering stages. First, a strong NLI model prunes atoms that are not entailed by the hypothesis. Second, atoms that are already entailed by the premise are removed, because δ-SNLI uses neutral premise-hypothesis pairs and the update should affect parts of the hypothesis.

• A human annotator then validates surviving atoms. Valid atoms must be grammatical, entail from the hypothesis, and not entail from the premise. Atoms that introduce new information or rely on pragmatic inferences are treated as invalid.

• In the validation sample, 95.7% of pruned atoms were judged valid. An external annotator labeled a sample of 100 atoms, yielding Cohen’s Kappa of 0.82.

• The authors further annotate atomic sub-problems as premise-atom-update tuples. Each atom is rated on a five-point scale from strongly weakens to strongly strengthens, with a midpoint for no effect. A second external annotator labeled 100 sample atoms, obtaining Kendall’s Tau of 0.79.

• The atomic label distribution shows that many δ-SNLI examples contain atoms with mixed polarity or no effect, highlighting the non-monotonic nature of defeasible inference.

• The paper uses the full δ-SNLI examples to benchmark models on the original binary strengthen-or-weaken task. It also uses the validated δ-SNLI-TEST atoms to evaluate whether models understand the atomic reasoning steps behind each full example.

• Because the original δ-SNLI training labels are binary while atomic evaluations include neutral effects, atomic accuracy is reported on non-neutral atoms, which account for about 80% of atoms.

• For the SNLI-TEST-SAMPLE analysis, the authors do not externally validate atom entailment. Instead, they measure consistency only on atomic sub-problems that the model itself treats as valid.

• No cropping strategy is used. Dataset construction instead relies on selecting neutral SNLI pairs, collecting or deriving crowd-written update sentences, applying NLI-based filtering to atoms, and adding human validity and effect annotations.

Method

The authors leverage a method that decomposes natural-language inference problems into elementary atomic propositions, validates those atoms, and then uses them as the basis for fine-grained evaluation of model consistency. The central idea is that a model’s overall judgment should align with its judgments about the smaller pieces of information that compose the hypothesis. This alignment can be checked for ordinary entailment tasks and, more importantly, for defeasible inference settings where an update strengthens or weakens a hypothesis.

Given a premise $P$P and a hypothesis $H$H, the method first generates an atomic decomposition of $H$H into atoms $a_1, \ldots, a_n$a1​,…,an​. Each atom represents a single piece of information in the hypothesis. In ordinary NLI, each atom defines a subproblem of the form $(P, a_i)$(P,ai​), where the model must determine whether $P$P entails, contradicts, or is neutral with respect to that atom. In defeasible NLI, the input also includes an update $U$U, and the corresponding atomic subproblem is $(P, a_i, U)$(P,ai​,U). The task for each atom is then to determine whether $U$U strengthens, weakens, or has no effect on $a_i$ai​.

The atomic decomposition module is built on Neo-Davidsonian event-based semantic representations. Sentences are treated as conjunctions of predicates over entities and event variables. For example, a sentence about a person performing an action can be represented in logical form as:

Each conjunct is mapped to a natural-language expression called an atom. This ensures that both the action and its semantic arguments appear as separate pieces of evidence. The authors use carefully hand-constructed exemplars to prompt a large language model, specifically llama-3-8b-instruct, to generate atoms for hypotheses.

Because atomic decompositions can introduce invalid or overly inferred facts, the method includes a validation workflow. For defeasible inference examples, non-monotonic reasoning makes validation especially important, because updates can change the inferential status of propositions in ways that do not follow strict monotonic entailment. The validation pipeline has two stages. First, a strong fine-tuned NLI model, based on DEBERTA-large, prunes atoms that are not entailed by the original hypothesis. Since the base premise-hypothesis pair is typically neutral, a second pruning step removes atoms that are already entailed by the premise. Second, human annotators validate the surviving atoms. A valid atom must be grammatical, entailed by the hypothesis, and not entailed by the premise. Atoms that introduce new pragmatic inferences are rejected.

For traditional NLI, the atomic atoms are used to define necessary logical consistency constraints between the overall premise-hypothesis label and the individual premise-atom labels. If $H$H is entailed by $P$P, then every valid atom must be entailed by $P$P:

If $H$H contradicts $P$P, then at least one valid atom must contradict $P$P:

If $H$H is neutral with respect to $P$P, then at least one valid atom must be neutral, and the other valid atoms may be neutral or entailed, but not contradictory. These rules give a diagnostic framework for checking whether a model’s global inference is supported by its local inferences.

In defeasible inference, the relationship between atoms and the overall judgment is softer. The update $U$U does not simply add logical premises. Instead, it shifts the plausibility of the hypothesis by affecting different atoms to different degrees. A human reading an update may judge that it strengthens the overall hypothesis because it supports some atoms strongly, weakens or leaves unchanged other atoms, and possibly combines several weak signals into a stronger overall effect.

The method therefore introduces the notion of a critical atom. A critical atom is the atom that the update most strongly affects and that aligns with the overall polarity of the defeasible judgment. If the update strengthens the hypothesis, the critical atom is the atom that is most strongly strengthened. If the update weakens the hypothesis, the critical atom is the atom that is most strongly weakened. Some examples have multiple critical atoms when several atoms are affected with comparable strength. This framing connects atomic inference to questions under discussion: each hypothesis can answer many possible questions, but a particular update tends to favor the question associated with the critical atom.

To group examples by the knowledge they evaluate, the method identifies semantically equivalent critical atoms. Each critical atom is embedded using NV-Embed-7B. Pairwise cosine similarity is computed, and a graph is constructed where nodes are critical atoms. An edge is added between two atoms if their embedding similarity exceeds a threshold, set to $\theta = 0.75$θ=0.75, and if the NLI validation model finds that the atoms bidirectionally entail each other. Maximal cliques in this graph are treated as clusters of equivalent critical atoms. Examples are then bucketed by these clusters, so that all examples sharing the same critical atom can be compared as different contexts for the same underlying inference.

As shown in the figure below:

The illustration shows two defeasible NLI examples that share the same critical atom, namely that the people are friends. The two examples differ in premise and update, producing different contexts for evaluating the same factual inference. In one context, the update supports the atom; in the other, it opposes the atom. The method uses this kind of grouping to test whether a model understands the critical fact itself or merely succeeds in specific premise-update contexts.

The inferential consistency metric formalizes this idea. Let $e_i$ei​ and $e_j$ej​ be two defeasible NLI examples that belong to the same critical atom bucket $B$B. Let $R_e$Re​ be a binary variable indicating whether the model’s prediction for example $e$e is correct, and let $\theta_B$θB​ denote the average correctness over examples in bucket $B$B. The inferential consistency $I_C$IC​ is defined as the probability that two examples sharing the same critical atom are either both predicted correctly or both predicted incorrectly:

Using bucket accuracies, this can be estimated as:

When a defeasible NLI example contains multiple critical atoms, the weight of that example is divided across all buckets corresponding to its critical atoms. This weighting prevents examples with multiple critical atoms from being counted disproportionately when measuring how consistently a model applies the same underlying fact across different contexts.

Experiment

Across natural language inference settings with atom-level annotations, experiments evaluate several large language models and baseline models on full examples, atomic subproblems, critical atoms, and cross-context consistency to examine reliability beyond raw accuracy. The findings show that accuracy can diverge from logical consistency, with strong models still producing inconsistent atomic and full-problem predictions and showing class-dependent consistency gaps. Models generally perform worse on all atomic inferences than on full examples, yet handle critical atoms better, suggesting they prefer direct, salient inferences while struggling with weaker or indirect reasoning. Inferential consistency is also limited, especially when different premises and updates require implicit, multi-hop, or background knowledge, indicating that correct full predictions do not necessarily reflect robust understanding.

The table presents instances from δ-SNLI and SNLI with hypothesis decompositions into atomic propositions and labels for the sub-problems involving those atoms. These examples show that hypothesis sentences can be broken into smaller pieces of information that are then evaluated against the premise, and in defeasible cases against an update. They also illustrate how validity depends on whether an atom is grammatical, entailed by the hypothesis, and not already entailed by the premise. Atomic decomposition separates a hypothesis into single pieces of information for more focused inference evaluation. In the defeasible inference examples, an update can strengthen, weaken, or leave unchanged the belief supported by a particular atom.

The evaluated LLMs achieve strong full-example accuracy and generally high logical consistency between their final predictions and atom-level subproblem labels. Consistency is highest when the full prediction is correct and drops substantially on incorrect examples, with model-dependent variation across predicted entailment, neutral, and contradiction labels. Although atomic decomposition is more interpretable, the induced atom-based label accuracy remains below full-example accuracy for all compared models. Full-example accuracy is strongest for the smaller GPT-4o model in the comparison. GPT-4o shows the highest overall logical consistency while having slightly lower full accuracy than GPT-4o-mini. Logical consistency on incorrect full examples is markedly lower than on correct examples. Neutral-label predictions are often more logically consistent than entailment-label predictions. Induced atomic-label accuracy does not outperform direct full-example accuracy.

The results compare model accuracy on full defeasible inference examples, all atomic sub-problems, and the critical atomic sub-problems that most directly drive each example. Models generally achieve higher accuracy on full examples and critical atoms than on all atomic sub-problems, suggesting that weaker or indirect effects are harder to evaluate. Accuracy on a full example is closely tied to whether the model predicts the critical atom correctly. Several recent models exceed the human benchmark on full examples. Models are stronger on critical atoms than on all atomic sub-problems. When the critical atom is predicted correctly, accuracy on the corresponding full example remains high. When the critical atom is predicted incorrectly, accuracy on the full example drops substantially.

The material explains that updates can act on the same hypothesis by targeting different atomic details, and that the most strongly affected atom identifies the question under discussion. Model analyses show room for improvement in inferential consistency, especially when contexts require implicit reasoning, multi-hop reasoning, or background knowledge. This critical atom framing helps reveal how context changes model behavior even when examples share the same underlying atomic fact. Updates may target different atoms of a hypothesis while leaving other derived atoms unchanged. Inferential consistency varies with context and can decline when updates demand harder reasoning or outside knowledge. Critical atoms provide a useful lens for judging whether models have internalized atomic facts across different examples.

The reported models show limited inferential consistency on defeasible NLI examples grouped by shared critical atoms, indicating that their predictions often vary across contexts. Performance ranges from about 74.5 for llama-3-8b-it to 87.0 for deberta-v3-large, leaving substantial room for improvement. deberta-v3-large has the highest inferential consistency among the evaluated models, while llama-3-8b-it has the lowest. Consistent failure or success across examples with the same critical atom is uncommon enough to suggest that models do not always robustly internalize the underlying facts. Inconsistent behavior appears when different contexts require varying amounts of implicit reasoning, multi-hop reasoning, or background knowledge.

The experiments evaluate large language models on full defeasible inference examples and atom-level subproblems, especially critical atoms that determine each example. The results show that full example accuracy is strong and closely tied to correct prediction of critical atoms, while atomic decompositions improve interpretability but do not produce more accurate labels than direct full example judgments. Logical consistency is high when full predictions are correct but weak on errors, and model behavior varies across contexts with the same critical atom, showing that implicit reasoning, multi-hop reasoning, and background knowledge remain difficult.