One-sentence Summary

Researchers from the University of Bern, Bern University of Applied Sciences, Stanford University, and colleagues introduce MultiLegalPile, a 689GB multilingual legal corpus spanning 24 languages across 17 jurisdictions that enables pretraining NLP models under fair use and establishes new state-of-the-art results on both the LEX-TREME and LexGLUE benchmarks.

Key Contributions

• The paper contributes MULTILEGALPILE, a 689GB legal pretraining corpus in 24 languages from 17 jurisdictions, curated from diverse legal data sources and supporting pretraining under fair use, with most data licensed permissively.
• Continued pretraining of XLM-R checkpoints yields two multilingual RoBERTa legal models, one multilingual Longformer for long documents, and 24 monolingual legal models trained on language-specific subsets.
• Continued pretraining establishes new state-of-the-art base and large results on LEX-TREME, with the Longformer reaching state-of-the-art in four tasks and the highest dataset aggregate score. Monolingual models set language-specific state-of-the-art in five languages by up to 11 macro F1, while English models reach state-of-the-art on five of seven LexGLUE tasks and the large model achieves the highest aggregate score.

Introduction

Large language models have made rapid progress on diverse benchmarks and even professional exams, but their training data remains heavily English-centric, web-sourced, and often legally unclear. This matters for specialized domains such as law, where legal texts are widely produced by public institutions and frequently available under permissive licenses, yet large multilingual legal pretraining corpora are scarce. Prior legal language models mostly focus on English or rely on corpora that are much smaller than what public legal systems can support.

The authors address this gap by curating MULTILEGALPILE, a large multilingual legal corpus covering 24 languages and 17 legal systems, mostly under permissive licenses. They continue pretraining XLM-R and a Longformer variant on this data to create LegalXLMs. Their models achieve new state-of-the-art results on LEXTREME and LexGLUE, showing that open, domain-specific, multilingual pretraining can substantially improve legal language understanding.

Dataset

• The authors use MULTILEGALPILE, a multilingual legal corpus intended for pretraining legal language models and released in XZ compressed JSONL format for efficient streaming.

• The corpus contains four main subsets:

  • Native Multi Legal Pile, 112 GB, compiled by scraping legal language sources from courts, administrations, and other public or open resources. The appendix describes 29 sources with differing licenses, from CC0 to CC BY-NC-SA.
  • Eurlex Resources, 179 GB, collected from EUR-Lex using the eurlex R package to generate SPARQL queries and download data, then converted with Python into a format suitable for large-scale processing.
  • Legal mC4, 106 GB, derived from the multilingual web crawl mC4 by filtering for legal content.
  • Pile of Law, 292 GB, an existing English-heavy legal corpus, mainly US focused, integrated into the final corpus.

• The Legal mC4 subset is built by applying regular expressions to mC4 to identify documents with legal references, such as statutory citations or article markers. The authors used legal citations as a signal for legal content, reviewed retrieved documents for five languages, and refined the regular expressions through multiple precision checks. This reduced the filtered corpus from 133 GB to 106 GB. The authors note that only about one in every thousand mC4 pages contains legal content.

• For the Native Multi Legal Pile, sources were collected in a broad, non-complete manner from available legal text collections. Data with inconsistent formats was converted into a common format, such as JSONL. Some sources required task-specific postprocessing, such as extracting text from XML tags.

• Language and text type are dominated by a few sources. English makes up more than half of the corpus because Pile of Law is integrated. Caselaw accounts for more than half of the corpus, partly because court rulings are widely available, especially in common law countries. Publicly available contracts make up less than 10% of the corpus.

• The authors preserve source-level license information in the dataset documentation. The overall MULTILEGALPILE release is described under CC BY-NC-SA, but individual subsets retain upstream licenses, including CC BY-NC-SA, CC0, CC BY, and ODC-BY for filtered Legal mC4.

• For evaluation, the paper uses external legal benchmarks such as LEXTREME and LexGLUE, covering tasks like legal classification, contract unfairness detection, topic labeling, named entity recognition, and court decision prediction. These benchmark datasets are not described as part of the MULTILEGALPILE training corpus.

• The supplied text does not specify training splits, mixture ratios, model-specific sampling weights, or cropping strategies for MULTILEGALPILE. It mainly describes corpus construction, filtering, format standardization, licensing, and intended use for legal NLP pretraining.

Method

The authors leverage a two-stage pipeline: first they build MULTILEGALPILE as a legal pretraining corpus, then they use it to continue pretraining transformer-based legal language models. The corpus stage converts all sources into xz-compressed JSONL. The source mixture is shown in the figure below.

For the web-derived component, the pipeline starts from mC4 and applies regular expressions to locate documents with legal markers, such as statute or ruling citations. The authors refine the regex rules through expert-assisted precision checks in several languages. This filtering is intentionally simpler than a learned classifier because regex execution is faster on a massive web corpus, while legal citations provide a high-signal indicator of legal text. Native legal materials are compiled by scraping public sources, normalizing their formats, and extracting text from structured containers such as XML. Eurlex data are generated through SPARQL queries and converted into the same processing pipeline, and the existing English legal corpus is integrated as an additional source.

The model stage trains Legal-XLM-R by recycling existing XLM-R transformer checkpoints. This initialization provides multilingual language knowledge before legal specialization. The authors also train a new 128K BPE tokenizer on the legal corpus. Tokens that overlap with XLM-R reuse the original embeddings, while newly introduced legal or language-specific tokens receive initialized embeddings. This design preserves general multilingual representations while increasing vocabulary capacity for legal terminology. The architecture remains a Transformer encoder with masked language modeling and shared input-output embeddings.

The legal corpus is strongly imbalanced across languages and sources, because the integrated English legal corpus contributes a large share of the total tokens. The language distribution is shown in the figure below.

This imbalance directly shapes the training process. The authors use a sentence sampler with exponential smoothing of sub-corpora sampling rates for both tokenizer training and continued pretraining, aiming to retain capacity for smaller languages and legal sources rather than letting majority English data dominate the updates. The models are trained in mixed case. Masked language modeling is used with an increased masking rate of 20% for base-size models and 30% for large-size models, and all predictions are made over masked tokens.

The continued pretraining uses batches of 512 sequences. The base-size model is trained for one million steps and the large-size model for 500,000 steps. The learning rate warms up linearly during the first 5% of total steps to $\eta_{\max}=10^{-4}$ηmax​=10−4, then decays under a cosine schedule. During the first 2.5% of training, the Transformer encoder is frozen and only embeddings are updated, which stabilizes the transition from general multilingual embeddings to a legal-specialized vocabulary.

For longer legal documents, the authors extend the base multilingual model to 4096-token contexts using windowed attention with a 128-token window. This long-context phase runs for 50,000 steps, uses a 15% masking probability, and sets the learning rate to $3\times10^{-5}$3×10−5 before decay. They also train 24 monolingual base-size models on language-specific subsets with a smaller 32K vocabulary and 200,000 steps, allowing multiple passes for low-resource languages and extended training for English.

Experiment

The experiments evaluate the pretrained legal encoders on two long-document legal NLU benchmarks, LEXTREME and LexGLUE, using task-appropriate setups and macro-F1 scores. Results show that legal-domain pretraining yields broad gains across multilingual classification, NER, and case-analysis tasks, with compact base models competitive with much larger general-purpose encoders and several models setting new state-of-the-art marks. The findings also indicate task and language sensitivities: long-document models perform best on some dataset-specific tasks but weaker across highly multilingual aggregates, monolingual models help low-resource languages, and contract-focused tasks benefit less when pretraining data contains fewer contracts.

The table summarizes prior domain-specific pretraining corpora and shows that legal corpora vary widely in size, from smaller English legal collections to much larger scientific and multilingual legal data. Compared with earlier legal corpora, the introduced legal pretraining corpus is notably larger and spans many more languages, supporting multilingual legal language modeling. The cited experiments indicate that models pretrained on this corpus are competitive with or better than general-purpose baselines, with especially strong gains on several legal tasks and low-resource languages. A multilingual legal pretraining corpus is larger and broader in language coverage than prior legal corpora. Pretrained legal models improve over general-purpose baselines on many tasks, though some tasks lag due to hyperparameters or domain coverage.

A two-round manual evaluation of regex filtered legal mC4 documents showed large differences across languages in the first round, with German and Spanish already highly precise and English much lower. Refining the regex expressions after the first review increased precision for English, French, and Italian while leaving the already strong German and Spanish results high. The improved filtering reduced corpus volume but made the retrieved legal content more specific. German and Spanish legal references were detected with consistently high precision English precision improved strongly after the regex expressions were refined French moved from moderate first-round precision to a perfect second-round result Italian precision rose from high to very high in the second evaluation

The model comparison shows that Legal-XLM-R variants are specialized legal encoders trained on a 24-language legal corpus, unlike general multilingual models trained on much broader corpora. They use fewer parameters and smaller vocabularies than several strong baselines, yet achieve benchmark results at or above larger general-purpose models, with the large legal model giving a small overall advantage. The legal Longformer is an exception in sequence handling, supporting much longer inputs than the other models. Legal-XLM-R-base reaches results comparable to a much larger general-purpose model. The legal models trade broad language coverage for legal-domain specialization across 24 languages. The legal Longformer supports much longer document inputs than the shorter-context models.

The aggregate macro-F1 results place the legal models very close to XLM-R-large, with the base model nearly matching the larger baseline despite using about one-third of its parameters. Legal-XLM-R-large is the strongest legal variant overall, while the language columns show that gains are uneven across settings. This suggests legal-domain pretraining can provide competitive aggregate performance without fully surpassing the large general multilingual model. Legal-XLM-R-base is competitive with XLM-R-large while using far fewer parameters. Legal-XLM-R-large leads among legal variants, but language-level improvements remain mixed.

The results compare legal-domain models with general-purpose encoders on legal classification tasks. Legal models reach new best results on ECtHR, SCOTUS, EUR-LEX, and CaseHOLD, while matching general-purpose encoders on LEDGAR and UNFAIR-ToS. Among the listed baselines, transformer models are far stronger than TFIDF+SVM, and RoBERTa-large is strongest in aggregate. Legal models set new best scores on ECtHR, SCOTUS, EUR-LEX, and CaseHOLD. LEDGAR and UNFAIR-ToS show weaker legal-model gains, possibly tied to limited contract coverage. RoBERTa-large has the highest aggregate among the listed general-purpose baselines.

Experiments evaluate a large multilingual legal pretraining corpus and Legal-XLM-R encoders against general-purpose multilingual baselines on legal classification and aggregation tasks. Data-quality checks show that refined filtering produces more precise legal content across languages, especially after iterative manual review, while preserving strong results for languages with initially high precision. Overall, legal-domain pretraining enables competitive or better performance than larger general models, with the base model achieving notable efficiency and the long-context variant supporting longer inputs, though gains remain uneven across languages and some contract-related tasks.