One-sentence Summary

Researchers from The University of Texas at Austin and Stanford University apply Distributed Interchange Interventions to language models, showing that these models converge on shared abstract analyses across English filler–gap constructions and reveal overlooked factors related to frequency, filler type, and context that could motivate changes to standard linguistic theory.

Key Contributions

• Causal interpretability methods, including Distributed Interchange Interventions, are applied to English filler-gap constructions to identify the internal mechanisms language models use for questions, relative clauses, and related structures.
• Transfer analyses show shared filler-gap mechanisms across constructions, with stronger transfer when constructions match in filler type, verb inversion pattern, information-structural fronting, or syntactic category.
• The results reveal overlooked frequency and animacy effects, plus weaker transfer between embedded and matrix clauses, suggesting refinements to syntactic accounts of constructional similarity.

Introduction

Large language models can process long-distance filler-gap constructions, such as wh-questions and clefts, which are important tests of whether models learn abstract syntax rather than only surface statistical patterns. Prior work showed that models are sensitive to these constructions, but results were mixed about whether processing is shared across constructions, and most evidence was behavioral rather than mechanistic. The authors leverage mechanistic interpretability, causal abstraction, and distributed alignment search to uncover filler-gap mechanisms in neural models and test whether those mechanisms transfer across different syntactic constructions. They find strong cross-construction generalization, shaped by linguistic similarity, lexical overlap, frequency, and animacy, while transfer does not readily extend across clause boundaries.

Dataset

• The authors use a template based text dataset rather than an existing corpus. They design sentential templates in the style of Arora et al. (2024) and sample many minimal pairs that differ in the targeted grammatical phenomenon. Table 1 provides the general template and example sentences.

• Target subset: the dataset covers seven filler gap constructions: embedded wh questions with a finite complementizer, embedded wh questions with a nonfinite complementizer, matrix level wh questions, restrictive relative clauses, clefts, pseudo clefts, and topicalization. For each construction, the authors create four templates.

• Target subset controls: templates vary in two ways, the animacy of the extracted object and the number of clausal boundaries between the filler and the gap, with one or two clauses. All target constructions extract a direct object from a verb phrase and follow a common template so that positions align across constructions.

• Control subset 1: subject verb number agreement. This control has a similar distance between the relevant words as the target constructions, but it does not require filler gap structure, because agreement is between clausemate elements.

• Control subset 2: continuation prediction after transitive or intransitive verbs. This control keeps the predicted label distribution similar to the target task. The authors preserve the distance between the minimal contrast and the prediction point by placing lexical items in faux contrast at the filler, article, and noun phrase positions.

• Data schema and processing: the data are constructed as position aligned text templates with slots such as filler, article, and noun phrase. No cropping, image preprocessing, or external corpus filtering is mentioned. The main processing step is generating sentence pairs or continuations from templates that isolate the intended grammatical contrast.

• Scale and use: the excerpt does not report raw example counts, train validation or test splits, or further filtering. The scale is defined by seven target constructions with four templates each, plus two control tasks. The authors use this data to compare model behavior on filler gap constructions with behavior on structurally similar but non filler gap controls.

Method

The authors leverage a two-part framework that first builds minimally contrasting linguistic stimuli and then uses causal interventions to test where a filler-gap mechanism is encoded. In the data construction stage, they generate controlled sentence sets for seven filler-gap constructions. For each construction, they design sentential templates that allow sampling many minimal pairs that differ only in the targeted grammatical property. Each construction uses four template variants that vary the animacy of the extracted object and the number of clausal boundaries between the filler and the gap left by extraction. All templates involve extraction of a direct object from a verb phrase, and the shared template structure enables alignment across constructions by token position. The design also includes control tasks, such as subject-verb number agreement and continuation prediction after transitive or intransitive verbs, to separate genuine filler-gap behavior from simpler linear proximity effects or label prediction heuristics.

The internal mechanism probing module uses Distributed Alignment Search. For a base input and a source input, the model produces hidden representations $\mathbf{b}$b and $\mathbf{s}$s from the same Transformer block and token position. The procedure learns a feature direction $\mathbf{a} \in \mathbb{R}^n$a∈Rn and applies the intervention

Here, $\mathbf{b}$b is the representation formed for the base example, while $\mathbf{s}$s is the corresponding representation formed for the source example that contains the feature to be transferred. The intervention moves the base representation along the learned feature axis while preserving orthogonal dimensions of $\mathbf{b}$b. The language model parameters are kept frozen, and only the vector $\mathbf{a}$a is learned through a standard cross-entropy objective. The training goal is to make the correct prediction after the intervention, for example inducing a gap prediction even when the base input does not originally encode the expected filler-gap structure.

The workflow supports within-class and cross-class transfer. Refer to the framework diagram below.

In the single-source setting, the feature direction for each construction is trained only on sentences sampled from templates belonging to that construction. In the leave-one-out setting, the feature direction for a construction is trained on templates from all other constructions, allowing the authors to test whether a mechanism learned for one class transfers to a held-out class. At each position from the filler onward and across every model layer, the training set contains 200 sampled sentences plus their corresponding minimal pairs, yielding balanced training sets of 400 sentences.

For evaluation, the authors measure how much more likely the counterfactual continuation associated with the source construction becomes after the intervention. They average the resulting counterfactual likelihood measure at each position-layer pair using evaluation sentences that do not overlap with the training set. Because the method is intended to localize the most causally relevant representation, scores are aggregated across layers by taking the maximum value at each position. This maximum-pooled score is then normalized using the corresponding score obtained on the training set, so that generalization to a held-out construction can be compared with generalization to the constructions used during training.

Experiment

Experiments used targeted interventions and transfer tests across filler gap constructions to examine whether language models share processing mechanisms and what factors drive that sharing. The first experiment showed broad shared representations across constructions, with stronger transfer when training and evaluation data overlapped or matched in animacy. The second experiment characterized constructions as sources or sinks, finding that generalization is shaped by construction frequency and linguistic similarity, so frequent constructions support less frequent ones while linguistically isolated constructions transfer poorly. The third experiment found limited generalization across clausal boundaries, with single-clause mechanisms transferring early in multi-clause sentences and again near the embedded verb, indicating that matrix and embedded clauses are handled by different mechanisms.

The presented construction set pairs single clause animate extraction variants with controls and linguistic parameters for several filler gap constructions. The entries vary by filler class, head child inversion, embedding phrase, and whether fronting is structurally required or discourse motivated. These contrasts provide controlled examples for testing whether a model represents each construction and whether interventions target the intended gap. Shared template positions support cross construction comparisons across different extraction types. The parameter block separates constructions by filler nature, inversion, embedding category, and fronting rationale. The agreement control keeps a comparable subject verb distance while omitting the filler gap dependency.

The regression results show that generalization in filler-gap processing is stronger when the training source and evaluated construction share linguistic properties. Shared filler type and shared embedding category are consistently associated with higher processing odds, while inversion and fronting status show positive effects in several positions. Linguistic similarity between constructions predicts stronger generalization Shared embedding properties yield strong positive effects Inversion and fronting effects are present but less uniformly strong

The experiments use controlled filler gap constructions that vary by filler type, inversion, embedding category, and fronting rationale to test whether models represent each construction and whether interventions target the intended dependency. Regression results show that generalization is strongest when the training and evaluated constructions share linguistic properties, especially filler type and embedding category. Inversion and fronting also show positive effects in some cases, but these are less consistent, while agreement controls provide comparable non dependency baselines.