One-sentence Summary

University of Edinburgh and Monash University propose INCLINE, an inference-time framework that bridges cross-lingual performance gaps in LLMs by learning alignment matrices via Least-Squares optimization on parallel sentences to transform low-performing language representations toward high-performing ones, significantly improving accuracy across nine benchmarks and five LLMs without costly retraining.

Key Contributions

• INCLINE learns alignment matrices from parallel source and target sentences through least-squares optimization and applies them during inference to transform low-performing language representations toward high-performing language spaces without fine-tuning the LLM.
• Evaluations across nine benchmarks, five LLMs, five discriminative tasks, four generative tasks, and 21 languages show that INCLINE significantly improves downstream performance, with average accuracy gains of up to +4.96 over strong baselines.
• Analyses show that INCLINE is cost-effective and broadly applicable, requiring minimal computational resources while remaining robust across LLM backbones, model sizes, and in-context learning settings; released code supports further research on underrepresented languages.

Introduction

Large language models excel at many NLP tasks, but they are often English-centric and show large performance gaps across languages, limiting access for low-resource users. Prior approaches rely on costly multilingual training data, large-scale fine-tuning, or prompting strategies that can be inconsistent and resource-intensive. The authors introduce INCLINE, an inference-time cross-lingual intervention that learns layer-wise alignment matrices from parallel sentences and maps low-performing language representations into a high-performing language representation space. This improves LLM performance across benchmarks without retraining or fine-tuning the model, offering a low-cost path to stronger multilingual capability.

Method

The authors design INCLINE as an inference-time cross-lingual intervention framework for improving large language model performance on source language tasks. Instead of fine-tuning the model, the method keeps the LLM parameters frozen and introduces lightweight alignment matrices that project source language hidden states into the representation space of a stronger target language. The pipeline has two core stages. As shown in the figure below:

The first stage learns cross-lingual alignment from parallel corpora. Given a parallel dataset $D = \{(\pmb{x}_i^{\mathrm{s}}, \pmb{x}_i^{\mathrm{t}})\}_{i=1}^{N}$D={(xis​,xit​)}i=1N​, where $\pmb{x}_i^{\mathrm{s}}$xis​ is a source language sentence and $\pmb{x}_i^{\mathrm{t}}$xit​ is its target language translation, the model extracts sentence-level hidden states at each layer by taking the representation of the last token. At layer $l$l, these representations are denoted as $\pmb{h}_{i,l}^{\mathrm{s}} \in \mathbb{R}^{d}$hi,ls​∈Rd for the source sentence and $\pmb{h}_{i,l}^{\mathrm{t}} \in \mathbb{R}^{d}$hi,lt​∈Rd for the target sentence, where $d$d is the hidden state dimension.

For each layer, the method learns a linear alignment matrix $\boldsymbol{W}_l$Wl​ that maps source representations toward target representations. The alignment objective is formulated as a least squares problem:

This objective minimizes the distance between the projected source sentence representations and the corresponding target sentence representations. Since the objective is a linear least squares problem, the optimal alignment matrix has a closed-form solution:

By fitting one such matrix per layer, the method constructs a layerwise mapping from source language representations to target language representation space. This reduces the cross-lingual representation gap without modifying the LLM weights.

The second stage applies the learned matrices during inference. For a test input $\pmb{q}^{\mathrm{s}}$qs in the source language, the method takes the hidden state of the last token at layer $l$l, denoted $\boldsymbol{h}_{q,l}^{\mathrm{s}}$hq,ls​, and projects it into the target language space:

The projected target language representation is then blended with the original source language representation through a residual addition:

The scalar $\alpha$α controls the intervention strength and balances the influence of the original source representation and the projected target representation. This formulation shifts the source language representation toward the target language space while preserving the original linguistic features. The modified hidden state $\boldsymbol{h}_{q,l}^{\mathrm{mix}}$hq,lmix​ is then used to continue the forward pass and generate the final output.

The intervention is performed as a minimal edit at the last token position of the input. Rather than modifying all token positions or replacing the model parameters, the method injects the transformed representation into a selected internal layer. This single-token residual update allows the LLM to leverage target language capabilities during inference while limiting perturbation to the rest of the representation.

Experiment

The experiments evaluate INCLINE across nine discriminative and generative multilingual tasks covering 21 languages, using BLOOMZ and several other LLM backbones alongside baselines such as machine translation, supervised fine-tuning, and activation intervention methods. They show that INCLINE provides efficient and meaningful performance gains in both task types by aligning non-English representations toward English, with hidden-state intervention offering the strongest effect. Additional analyses indicate that the method is computationally low-cost, improves cross-lingual prediction consistency, generalizes across models, scales with model size, complements in-context learning, benefits from higher-quality parallel data, and requires language-dependent intervention strength.

Across discriminative tasks evaluated by accuracy, INCLINE substantially improves performance over the baseline and lightweight activation intervention methods while avoiding costly fine-tuning or third party translation dependencies. Strong machine translation baselines can perform well on some tasks, but the paper frames them as higher cost compared with representation alignment methods. The reported aggregates also show clearer performance for seen languages than for unseen languages. INCLINE produces larger discriminative gains than simpler intervention methods while remaining efficient at inference time. MT-GOOGLE is competitive on several discriminative averages, but its strength comes with external machine translation cost. Seen languages generally achieve higher macro accuracy than unseen languages across discriminative tasks.

Across the generative tasks, MT-GOOGLE shows the strongest reported performance on relation extraction and math, although translation results are missing for it. The intervention methods provide only limited overall gains, with ITI especially weak on machine translation and CAA mostly offering modest changes relative to the baseline. The reported INCLINE comparisons indicate broader generative improvements than the baseline and intervention methods, with a notable gain on math accuracy. MT-GOOGLE has the best reported scores on MZsRE and MGSM but no reported Flores or WMT23 scores. ITI greatly degrades machine translation quality while only slightly improving math accuracy. CAA improves relation extraction over the baseline but leaves math nearly unchanged and gives mixed translation performance.

Across multilingual tasks, the INCLINE variants generally improve over the baseline, with hidden state intervention producing the largest overall gains. Intervening on attention outputs, feed-forward blocks, and embeddings can also improve performance, but the gains are more variable across tasks. Hidden state intervention gives the most consistent improvement among the tested INCLINE variants. Attention and feed forward interventions remain competitive, while embedding intervention is more task dependent.

Across five LLM backbones on XStoryCloze, INCLINE produces average accuracy gains over the corresponding baseline models. The consistent improvements suggest that INCLINE is broadly effective across different model families rather than tied to a single architecture. INCLINE improves average accuracy for every evaluated backbone relative to its baseline. The magnitude of improvement varies by model, with some instruct and chat models showing especially large gains. Language level results show broad improvements, including notable gains in lower scoring baseline languages for some models.

The table compares baseline, INCLINE, and INCLINE-FDEV BLEU scores across nine languages. INCLINE improves over the baseline across the reported languages and the average score, while INCLINE-FDEV provides an additional gain, indicating that using higher quality parallel sentences for alignment training is beneficial. INCLINE consistently raises BLEU relative to the baseline across the reported languages. INCLINE-FDEV outperforms both the baseline and standard INCLINE, supporting the benefit of high-quality parallel training data. Average BLEU increases from baseline to INCLINE and further to INCLINE-FDEV.

The experiments evaluate INCLINE against baselines and lightweight activation interventions on discriminative, generative, cross-model, and translation tasks. INCLINE yields consistent multilingual gains without fine-tuning or external translation cost, with hidden-state alignment most effective and larger benefits for seen languages and higher-quality parallel data. Although machine translation baselines can excel on some generative tasks, they are more costly and uneven, while activation interventions offer only limited or task-dependent improvements. Overall, INCLINE appears robust across tasks and model families, improving accuracy and translation quality relative to lightweight baselines.