One-sentence Summary

Researchers from Tsinghua University and Microsoft propose LLMLingua-2, a task-agnostic prompt compression method that formulates compression as token classification using a Transformer encoder trained on a distilled extractive dataset, replacing entropy-based causal LLaMa-7B selection with bidirectional context to achieve $3x-6x$3x−6x faster compression while maintaining faithful, generalizable performance across LLMs on benchmarks like Meeting-Bank and GSM8K.

Key Contributions

• The paper introduces a data distillation procedure that derives knowledge from an LLM (GPT-4) to compress prompts without losing essential information, and publicly releases an extractive text compression dataset built from MeetingBank source texts paired with their compressed versions.
• The paper formulates prompt compression as a token classification task in which each token is labeled preserve or discard, using a Transformer encoder to capture essential information from full bidirectional context while relying on small models such as XLM-RoBERTa-large and mBERT to achieve lower latency and guarantee faithfulness of the compressed prompt to the original content.
• Results show significant performance gains over strong baselines across in-domain and out-of-domain benchmarks, robust generalization across LLMs from GPT-3.5-Turbo to Mistral-7B, and compression speeds 3x to 6x faster than existing methods, with end-to-end latency accelerated by 1.6x to 2.9x at compression ratios of 2x to 5x.

Introduction

Recent advances in techniques like chain-of-thought prompting and retrieval augmented generation have driven LLM prompts to tens of thousands of tokens, leading to higher computational costs, increased latency, and degraded information perception. Prompt compression addresses these issues, but existing task-aware methods sacrifice generalizability by relying on task-specific features, while task-agnostic approaches that use information entropy from causal small language models face two major challenges: the entropy metric is not aligned with the compression objective, and unidirectional context fails to capture all essential information. Furthermore, current compression datasets are either abstractive, risking hallucinated content, or extractive and lacking in detail for downstream tasks.

The authors tackle these problems by first proposing a data distillation procedure that leverages GPT-4 to create a new extractive compression dataset from MeetingBank, preserving crucial information. They then reformulate prompt compression as a token classification task where a Transformer encoder predicts whether each token should be preserved, effectively using full bidirectional context and guaranteeing faithfulness to the original content. Their model demonstrates significant performance gains over strong baselines on both in-domain and out-of-domain benchmarks, while being 3x to 6x faster than existing compression methods and achieving 1.6x to 2.9x end-to-end latency improvements with compression ratios of 2x to 5x.

Dataset

Dataset Overview

The dataset is built from two primary sources: original texts and their compressed counterparts generated by an LLM. To create high-quality training data, the authors design a three-stage pipeline that includes data distillation, data annotation, and quality control.

1. Data Distillation

• The authors use GPT-4 to generate compressed versions of original texts.
• The goal is for the compressed prompts to meet three criteria: token reduction, informativeness, and faithfulness.
• The instruction explicitly tells GPT-4 to discard unimportant words from the original text without adding any new words, to reduce hallucinated content.
• A fixed compression ratio is not used. Instead, GPT-4 is prompted to compress the original text as short as possible while retaining as much information as possible.
• To maintain performance, long contexts are segmented into chunks of no more than 512 tokens, ending with a period. Each chunk is compressed individually to avoid high compression ratios that cause information loss.

2. Data Annotation

• The pairs of original and compressed texts are used to create token-level labels.
• Each token in the original text is assigned a binary label to indicate whether it should be preserved or discarded after compression.
• The annotation process addresses three main challenges: ambiguity (a word may appear multiple times in the original), variation (GPT-4 may alter tense or plural forms), and reordering (the order of words may change in the compressed version).

3. Quality Control

• Two metrics are introduced to filter low-quality examples.
• Variation Rate (VR): Measures the proportion of words in the compressed text that are absent from the original text. A higher rate suggests a higher chance of hallucinated content, so the top 5% highest VR examples are excluded.
• Alignment Gap (AG): Measures the quality of the automatically annotated labels. It is defined as the difference between the hitting rate (the proportion of words in the compressed text found in the original) and the matching rate (the proportion of original words correctly labeled as kept). A perfect annotation has an AG of 0. The top 10% highest AG examples are discarded.

The resulting filtered dataset is used to train a model for prompt compression, where the binary labels serve as supervision signals for deciding which tokens to keep.

Method

The proposed method establishes a pipeline for prompt compression that integrates a data distillation process, a rigorous quality control mechanism, and a token classification model. The framework begins with extracting knowledge from a large language model (GPT-4) to generate compressed texts. To ensure faithfulness, the instruction design explicitly restricts the model to discarding unimportant words from the original text without adding any new content. Additionally, to accommodate varying information densities across different text genres, fixed compression ratio restrictions are removed from the instructions, allowing the model to compress text as concisely as possible while retaining essential information.

However, the authors observe that GPT-4 tends to apply aggressive compression ratios when processing very long contexts, which leads to substantial information loss.

As shown in the figure below, there is a clear correlation where the compression ratio increases with the length of the context. To mitigate this issue, the distillation procedure segments long contexts into multiple chunks, each containing no more than 512 tokens and ending with a period, before compressing them individually.

Following the generation of compressed text pairs, the system proceeds to data annotation, assigning a binary label (preserve or discard) to each token in the original text. This process addresses challenges such as word ambiguity, variation in word forms, and reordering. To ensure dataset quality, two metrics are employed. The Variation Rate (VR) measures the proportion of words in the compressed text that are absent in the original text, serving as an indicator of hallucinated content. It is defined as:

Examples with the top 5% highest VR scores are excluded. Furthermore, the Alignment Gap (AG) evaluates the quality of the annotations by comparing the Matching Rate (MR) and the Hitting Rate (HR). The AG is calculated as:

Examples with the highest 10% alignment gap values are discarded to maintain high annotation quality.

The core compression module is formulated as a binary token classification problem. The architecture utilizes a Transformer encoder $f_\theta$fθ​ as the feature extractor to capture bidirectional context information, followed by a linear classification layer. Given an original prompt $x = \{x_i\}_{i=1}^N$x={xi​}i=1N​, the feature vectors are computed as:

The probability distribution for the $i$i-th token is then predicted as:

where $\Theta = \{ \theta, W, b \}$Θ={θ,W,b} represents the trainable parameters. The model is trained using the cross-entropy loss function:

where $y_i$yi​ denotes the ground truth label for the $i$i-th token.

During inference, the compression strategy follows a three-step process to achieve a target compression ratio $1/\tau$1/τ. First, the target number of tokens to be preserved is calculated as $\tilde { N } = \tau N$N~=τN. Next, the model predicts the preservation probability $p_i$pi​ for each token. Finally, the system retains the top $\tilde { N }$N~ tokens with the highest probabilities, maintaining their original order to form the compressed prompt. This approach ensures low latency and faithfulness to the original content.

Experiment

The experiments evaluate the LLMLingua-2 prompt compressor, trained on MeetingBank data, across in-domain summarization and QA tasks plus out-of-domain long-context, reasoning, and in-context learning benchmarks using GPT-3.5 and Mistral-7B as target LLMs. The method outperforms task-agnostic baselines while remaining competitive with the original prompt, though it lags behind task-aware compression like LongLLMLingua on long-context tasks. Additional findings highlight its low computation overhead, effective context-aware information preservation, reversible compression for prompt reconstruction, and the importance of the instruction and chunk-wise compression design.

LLMLingua-2 and its smaller variant outperform prior compression baselines on MeetingBank, achieving substantially higher QA exact match and summary scores while using far fewer tokens than the original prompt. Their performance approaches that of the uncompressed original prompt, indicating effective compression with minimal information loss. LLMLingua-2 improves QA exact match by about 19 points over LLMLingua and over 20 points over Selective-Context. LLMLingua-2 achieves summary RougeL and BERTScore close to the original prompt, despite compressing token count by roughly 3x. The smaller LLMLingua-2-small model performs nearly on par with the full model, showing strong compression with a lightweight encoder.

On out-of-domain long-context benchmarks, task-aware compression methods such as LongLLMLingua achieve the highest average scores, while the proposed task-agnostic model LLMLingua-2 outperforms other task-agnostic baselines and can match or exceed the original prompt in some cases. However, it still lags behind task-aware methods that leverage question information. LongLLMLingua, a task-aware method, attains the best average performance across LongBench and ZeroSCROLLS under the 2,000-token constraint. The task-agnostic LLMLingua-2 surpasses other task-agnostic baselines and shows comparable or slightly better results than the original prompt in certain scenarios. The performance gap versus task-aware compression is attributed to the additional question-derived information those methods exploit.

LLMLingua-2 and its smaller variant match or exceed the performance of the original full prompt on reasoning tasks while using far fewer tokens, and they outperform the Selective-Context and LLMLingua baselines. The improvements are consistent across both GSM8K and BBH under different shot constraints, with the full LLMLingua-2 model showing the best trade-off between accuracy and compression. LLMLingua-2 achieves accuracy comparable to the full prompt on GSM8K while reducing token usage by roughly 5x to 14x. On BBH with the half-shot constraint, LLMLingua-2 slightly outperforms the full prompt while using only a fraction of the tokens. The smaller LLMLingua-2-small variant performs nearly as well as the larger model, showing that a compact compressor can maintain strong reasoning performance.

With Mistral-7B as the target model, LLMLingua-2 and its smaller variant achieve higher summarization and QA scores than both baseline compression methods and the original uncompressed prompt, while using far fewer tokens. The performance gain suggests that shorter, information-dense prompts can improve Mistral-7B's inference quality, likely because it handles long contexts less effectively. LLMLingua-2 and LLMLingua-2-small outperform Selective-Context and LLMLingua on both MeetingBank and LongBench tasks. Compressed prompts from LLMLingua-2 yield better results than the original prompt, despite using roughly 3x fewer tokens. The smaller LLMLingua-2-small model achieves near-identical performance to the full version, showing efficiency at a reduced scale.

On MeetingBank, LLMLingua-2 achieves much lower latency than other compression baselines across all compression ratios. Its end-to-end speedup ranges from 1.6x to 2.9x, and it also reduces GPU memory costs by 8x, indicating a notably smaller computational overhead. LLMLingua-2 latency is consistently lower than Selective-Context and LLMLingua across 2x, 3x, and 5x compression ratios. End-to-end speedup from LLMLingua-2 increases with compression ratio, reaching up to 2.9x at 5x compression. LLMLingua-2 incurs minimal additional latency growth as compression ratio increases, unlike other baselines which show higher delays.

The experiments evaluate LLMLingua-2, a task-agnostic prompt compression method, against baselines like LLMLingua and Selective-Context across meeting QA, summarization, reasoning, and out-of-domain long-context benchmarks. LLMLingua-2 and its smaller variant consistently match or exceed the performance of the original uncompressed prompt while using substantially fewer tokens and clearly outperform the compression baselines. On reasoning tasks and with Mistral-7B as the target model, compressed prompts even improve over the original prompt, suggesting that shorter, information-dense inputs benefit models with weaker long-context handling. Although task-aware methods that exploit question information still lead on some long-context benchmarks, LLMLingua-2 also reduces latency and GPU memory costs, and its compact small version performs nearly on par with the full model.