One-sentence Summary

Researchers from University of Washington, Allen Institute for AI, and Stanford University present INFINI-GRAM MINI, an FM-index-based system that makes petabyte-level Internet text corpora searchable with indexes occupying only $44\%$44% of corpus size, achieves $18\times$18× faster indexing and $3.2\times$3.2× lower memory use than prior implementations, and reveals up to $74.2\%$74.2% contamination in GSM8K and other core language model benchmarks.

Key Contributions

• INFINI-GRAM MINI is a full-text search system built on the FM-index that stores indexes at 44% of corpus size while supporting exact string counting and document retrieval in massive text corpora. Its compressed suffix-array representation enables Internet-scale search with much lower storage overhead than prior exact-match search implementations.

• An optimized FM-index implementation improves indexing speed by 18 times and reduces indexing memory use by 3.2 times, while query memory becomes negligible through an on-disk SDSL index. Experiments show indexing 83TB of Internet text in 99 days on one 128-vCPU node, or in 19 hours using 137 such nodes.

• INFINI-GRAM MINI enables large-scale benchmark contamination analysis, revealing heavy overlap between Internet crawls and core language model evaluation benchmarks, including contamination up to 74.2% in GSM8K. It also releases a contamination bulletin, web interface, API endpoint, and source code to enable public search and ongoing contamination monitoring.

Introduction

Modern language models are trained on very large text corpora such as Common Crawl, so the ability to search exact long sequences in this data is important for understanding model behavior, curating datasets, and auditing benchmark contamination. Earlier exact-match search systems rely on suffix arrays, suffix automata, or proprietary engines, but their indexes are often several times larger than the original text, making them impractical for petabyte-scale training corpora. The authors leverage the FM-index, a compact full-text index common in bioinformatics, to build a much smaller searchable index, with practical storage around 0.44 of the corpus size and as low as 0.26. They introduce INFINI-GRAM MINI, a parallelized system for constructing and querying the FM-index at scale, which indexed 83 TB of pretraining and web text and supports fast occurrence counting and document retrieval. Using this tool, they find heavy exact-match contamination in several standard benchmarks, showing that inflated evaluation results can arise from web-scale training data.

Dataset

• The authors use indexed text corpora rather than a conventional supervised training dataset. They build indexes for three corpora: The Pile, DCLM-baseline, and Common Crawl from January to July 2025. A single 8.7 GB DCLM-baseline file is also used for timing and memory experiments.

• The corpus is stored as a concatenated byte sequence. Each document is encoded in UTF-8, and documents are joined using the byte 0xFF, which is not used by UTF-8, as a document-boundary marker. Working with bytes keeps the alphabet size small at 256.

• Alongside the text, the authors store a text offset file that records the starting position of each document. This supports retrieval of whole documents and associated metadata. Document metadata is also indexed and stored in a compressed searchable form.

• For large corpora, the data is partitioned into independent shards. Searching across shards yields the same results as searching a single larger index, enabling distributed construction and retrieval under limited RAM.

• The indexing runs use shards up to 700 GB on nodes with 128 vCPUs and 2 TiB RAM. Each such shard can be indexed in about 12 to 19 hours. If the full Common Crawl dataset is considered, its approximate size is 1 PB, which would be split into about 1,500 shards.

• The constructed indexes are compressed. The authors sample the suffix array every 32 entries and the inverse suffix array every 64 entries, resulting in index sizes about 0.44 times the original corpus size.

• The paper does not describe source-specific filtering rules, train or validation splits, mixture ratios, or cropping strategies. The data processing is mainly document concatenation, UTF-8 byte encoding, boundary marking, offset tracking, metadata indexing, and sharded corpus construction.

Method

The authors design the indexing system around the FM-index, a compressed suffix-array-based structure that can count and locate patterns without explicitly storing the full text. The architecture has two tightly connected data-structure layers: a sampled suffix array plus inverse suffix array, and a Burrows-Wheeler transform compressed with a Huffman-shaped wavelet tree. Refer to the example diagram below:

The suffix array $SA$SA stores the starting positions of all suffixes of a text $T$T in lexicographic order. Formally, $SA[i] = j$SA[i]=j when the suffix $T[j \ldots n]$T[j…n] is the $i$i-th suffix in sorted order. Storing the full suffix array is expensive, so the implementation samples it at regular intervals, keeping only every $a$a-th entry. When an unsampled suffix-array value is needed, the system recovers it through auxiliary index structures. The inverse suffix array $ISA$ISA, defined by $ISA[j] = i$ISA[j]=i if $SA[i] = j$SA[i]=j, is also sampled, often with a sparser sampling rate because it is used mainly to restart reconstruction from a known text position.

The Burrows-Wheeler transform, or BWT, reorders the text into a transformed string $L$L. Each position $i$i in $L$L stores the symbol preceding the suffix ranked $i$i in the suffix array:

with the termination symbol used otherwise. The important design insight is that the BWT clusters equal symbols and preserves enough structure to support backward traversal through the text. This is governed by the Last-to-First mapping:

where $c$c is the symbol at position $i$i in $L$L, $C[c]$C[c] is the number of symbols lexicographically smaller than $c$c, and $\mathrm{rank}(c,i)$rank(c,i) counts occurrences of $c$c up to position $i$i. The LF mapping enables the system to extend a partial pattern backward by repeatedly moving from $L$L to the first-column ordering.

To compress $L$L, the system stores it in a Huffman-shaped wavelet tree. Each leaf represents one alphabet symbol, while internal nodes store bitvectors that record whether each position belongs to the left or right subtree. The Huffman-shaped shape prioritizes frequent symbols so that they receive shorter codes. This tree also supports the two operations required by the LF mapping: $\mathrm{rank}(c,\ell)$rank(c,ℓ) counts occurrences of $c$c in the prefix $L[0 \ldots \ell-1]$L[0…ℓ−1], and $\mathrm{select}(c,m)$select(c,m) returns the position of the $m$m-th occurrence of $c$c in $L$L. Both operations are designed to run in time proportional to the entropy of the transformed text.

The construction pipeline first converts documents into raw UTF-8 bytes rather than tokens. Each document is represented as a byte string, and document boundaries are marked with a byte value outside normal UTF-8 encoding, specifically $\texttt{\textbackslash xff}$\xff. These byte sequences are concatenated into a single large text, allowing the FM-index to operate on one global string while preserving document boundaries through a separate offset file. The offset file records where each document begins, which enables retrieval of full documents and their associated metadata. Metadata can also be indexed in compressed form so that it remains searchable.

The system partitions very large corpora into shards and builds an independent FM-index for each shard. This makes construction embarrassingly parallel and keeps memory requirements bounded on each worker node. During indexing, the major steps, including suffix-array construction, BWT construction, wavelet-tree construction, suffix-array sampling, and inverse-suffix-array sampling, are parallelized. The implementation samples the suffix array every $a$a entries and the inverse suffix array every $b$b entries, with practical settings balancing index size against query latency. In one reported configuration, the resulting index is about $0.44$0.44 times the size of the corpus, while more aggressive sampling can reduce the index further toward a lower bound related to the entropy of the compressed text.

At query time, the system supports two exact-match operations: counting occurrences of a query string and retrieving documents that contain it. Index files are stored on disk and accessed through memory-mapped files, so the query process requires minimal resident RAM. For counting, the find operation traverses the LF mapping over the query bytes. If the corpus is divided into $k$k shards, the query is dispatched to all shards in parallel, and the final count is the sum of shard-level counts. The number of random disk reads is proportional to $O(k |Q| H_0)$O(k∣Q∣H0​), where $Q$Q is the query length and $H_0$H0​ is the zeroth-order entropy of the BWT.

For document retrieval, the pipeline begins with the same find operation to obtain a range in the suffix array corresponding to the query string. Each entry in that range identifies one occurrence of the query. The locate operation then maps each suffix-array position back to an original text position using sampled suffix-array values and rank/select queries on the BWT. Once the original position is known, the system binary searches the text offset file to determine the enclosing document boundaries. Finally, the reconstruct operation uses the sampled inverse suffix array and BWT to recover the document text around the match. Reconstruct is parallelized by dividing long document segments into chunks, and the whole retrieval stage can be further parallelized across multiple matching documents. The cost of locate and reconstruct grows with the sampling rates and the length of the retrieved text, so latency depends strongly on disk random-access throughput.

Experiment

The experiments evaluate INFINI-GRAM MINI’s exact-match querying workflow, showing that disk-backed counting and document retrieval can operate with modest memory and second-level latency despite compressed-index overhead. Using this system, the authors scan widely used LLM benchmarks against major text corpora and find non-trivial contamination, especially in older knowledge, commonsense, and reading-comprehension benchmarks, with larger or newer corpora often producing more contaminated entries. Further document-level inspection reveals that many contaminated cases contain the benchmark question and answer verbatim, indicating a risk of memorization rather than true reasoning, and the work extends these findings into an ongoing monitoring system for evolving corpora and new benchmarks.

INFINI-GRAM MINI can index text corpora ranging from very small validation data to multi-terabyte web crawls, producing compressed indexes that are consistently smaller than the original corpora. Indexing time and shard count increase with corpus size, and partitioning enables parallel index construction while preserving equivalent search results. The reported setup also uses byte-level concatenation, document offsets, and searchable metadata to support whole-document retrieval. Index sizes are roughly half or less of original dataset sizes across corpora. Larger corpora require more CPU node-days and are divided into more shards. Parallelized indexing steps substantially reduce time and peak memory compared with single-threaded SDSL baselines.

The results show substantial benchmark contamination across multiple training corpora, especially against the large DCLM-baseline corpus. Widely used reasoning and knowledge benchmarks such as MMLU, MMLU-Pro, ARC-Challenge, and ARC-Easy show high dirty-entry rates, suggesting some reported model capabilities may be inflated by exposure to evaluation content. Contamination is not uniform: it increases with corpus scale and source diversity, while most dirty entries contain exact question-answer matches rather than only superficial overlaps. DCLM-baseline generally shows higher contamination than Pile-train, with DCLM-baseline producing markedly more dirty entries for many benchmarks. MMLU, MMLU-Pro, ARC-Challenge, and ARC-Easy exhibit especially high contamination levels against DCLM-baseline. Common Crawl crawl results vary by crawl, and some single-crawl checkpoints show lower dirty rates than DCLM-baseline, likely because DCLM-baseline is larger and more diverse. Most dirty entries are exact matches of benchmark questions and answers, although paraphrased answers and question-only matches also contribute to contamination risk.

The examples show several ways benchmark items can leak into training corpora, including exact question and answer matches, natural language answers that reveal the benchmark response, question only exposure, and false positives from superficially similar text. Exact question and answer matches are the most frequent contaminated pattern across the corpora, while question only and paraphrased answer cases are less common but can still support answer retrieval. Short benchmark questions are especially prone to false positive matches. Exact benchmark question and correct answer matches are the dominant form of contamination in the examined corpora. Paraphrased or question only text can still reduce evaluation difficulty by helping models infer or retrieve the correct response.

Experiments show that INFINI-GRAM MINI scales from small validation data to very large web corpora, producing compressed indexes while maintaining searchable document retrieval, and that partitioning and parallel construction improve efficiency without changing search behavior. Additional analyses reveal substantial benchmark contamination in widely used training corpora, with larger and more diverse corpora such as DCLM-baseline leaking more evaluation content than smaller sources, especially for reasoning and knowledge benchmarks. Qualitative case studies indicate that exact question-answer matches dominate contamination, though shorter questions, paraphrased answers, and question-only exposure can also inflate model performance.