One-sentence Summary

Researchers from Microsoft Research Asia, Harbin Institute of Technology, and Beihang University propose LayoutLM, a pre-training framework that jointly models text, layout, and visual features from scanned document images, achieving new state-of-the-art results on form understanding ($70.72 \to 79.27$70.72→79.27), receipt understanding ($94.02 \to 95.24$94.02→95.24), and document image classification ($93.07 \to 94.42$93.07→94.42).

Key Contributions

• LayoutLM is the first pre-training framework to jointly model text and layout information from scanned document images in a single architecture.
• The model incorporates image embeddings to capture the visual appearance of words, enabling joint modeling of textual, layout, and visual signals.
• LayoutLM achieves new state-of-the-art results on form understanding (F1 from 70.72 to 79.27), receipt understanding (F1 from 94.02 to 95.24), and document image classification (accuracy from 93.07 to 94.42).

Introduction

Documents are critical to business operations, yet extracting structured information from the varied layouts and formats of invoices, receipts, forms, and other business documents typically relies on costly manual effort or brittle rule-based systems. Prior deep learning methods for document understanding have focused on detecting individual regions with models pre-trained on either vision or language tasks, but they are limited by scarce labeled data and a lack of joint modeling of textual content and layout structure. The authors introduce LayoutLM, a self-supervised pre-training approach that combines textual tokens, 2‑D position embeddings representing spatial layout, and image embeddings from scanned documents. By jointly pre-training on large-scale unlabeled document images with a masked visual‑language model and a multi‑label document classification objective, LayoutLM captures the interplay between text and layout, leading to significant improvements on downstream tasks such as form understanding and receipt information extraction.

Dataset

The authors assemble a pre-training corpus from the IIT-CDIP Test Collection 1.0, which spans over 6 million documents and more than 11 million scanned page images. Originally, each document supplies OCR-extracted text and XML metadata (unique ID, noisy labels). To capture the 2D layout needed by LayoutLM, they re‑process every image with Tesseract OCR, extracting both words and their bounding boxes, and save the results in the hierarchical hOCR format. This creates a pre-training set of document images paired with word‑level spatial positions.

Three fine‑tuning datasets are used:

• FUNSD – 199 noisy scanned forms (149 train, 50 test) containing 9,707 semantic entities annotated as question, answer, header or other, together with inter‑entity links. Evaluation metric: word‑level F1.
• SROIE (Task 3) – 626 training and 347 testing receipt images, with text lines and bounding boxes labelled for company, date, address and total. Evaluation metric: exact‑match entity F1.
• RVL‑CDIP – 400,000 grayscale images in 16 classes (letter, form, invoice, etc.), split into 320k train / 40k val / 40k test. Images are resized so the largest dimension does not exceed 1,000 pixels. Evaluation metric: overall classification accuracy.

During pre‑training, the model consumes the re‑processed IIT-CDIP images with hOCR‑derived positions; fine‑tuning uses each dataset’s own bounding‑box annotations and labels.

Method

The authors introduce LayoutLM, a model that jointly leverages text, layout, and visual information for document understanding tasks. Built upon the BERT architecture, LayoutLM extends the standard input representation to incorporate spatially aware and image-level signals, enabling richer semantic modeling of visually rich documents.

The core idea stems from the observation that BERT-like models, despite their strong performance on plain-text NLP problems, ignore the geometric arrangement and visual appearance of text on a page. Two types of information are particularly valuable: document layout, which captures the relative 2-D positions of words, and visual features, which encode font styles, emphasis, and the overall page structure. LayoutLM embeds both into the pre-training framework while maintaining the BERT backbone, allowing the model to attend jointly over linguistic and visual cues.

In the LayoutLM architecture, the input representation of a token sums the standard word embedding and segment embedding with two additional components. The first is a 2-D position embedding that models the spatial location of each word. A document page is treated as a coordinate system with the top-left corner as the origin, and each word's bounding box is specified by $(x_0, y_0, x_1, y_1)$(x0​,y0​,x1​,y1​). Four separate embeddings are learned for the x and y coordinates of the top-left and bottom-right corners, with the x-coordinate embeddings sharing one table and the y-coordinate embeddings sharing another. This design injects relative positional signals directly into the Transformer's self-attention mechanism, aligning layout with textual semantics.

The second addition is an image embedding derived from visual features. Using OCR-provided bounding boxes, the page image is partitioned into word-level regions, and a pre-trained Faster R-CNN model extracts a visual feature vector for each region, producing a token-level image embedding. The same Faster R-CNN model also generates a whole-image feature for the special [CLS] token, providing a document-level visual representation. These image embeddings are combined with the textual and position embeddings before being fed into the multi-layer bidirectional Transformer encoder.

Pre-training LayoutLM involves two self-supervised objectives. The primary task, Masked Visual-Language Model (MVLM), randomly masks some input tokens while keeping their 2-D position embeddings intact. The model must predict the masked tokens from the surrounding textual context and the corresponding layout coordinates, forcing it to fuse language and spatial information. In this way, MVLM closes the gap between visual and language modalities. An optional second task, Multi-label Document Classification (MDC), leverages document tags from the pre-training corpus to learn document-level representations. When available, these tags serve as supervision to help the model cluster knowledge from different domains and generate a high-quality document embedding; the MDC loss can be omitted when labels are not present.

After pre-training, LayoutLM is fine-tuned on downstream document image understanding tasks. For form understanding and receipt understanding, the model predicts a sequence of BIOES tags (Begin, Inside, End, Single, Outside) for each token, performing token-level sequential labeling to extract entities such as keys and values. For document image classification, the representation of the [CLS] token is used to predict the document category. The end-to-end fine-tuning updates all parameters, adapting the pre-trained multimodal representations to the specific structure and semantics of each task.

Experiment

The evaluation pre-trains LayoutLM on the large-scale IIT-CDIP document collection and fine-tunes on three tasks: form understanding (FUNSD), receipt information extraction (SROIE), and document image classification (RVL-CDIP). The experiments demonstrate that incorporating 2-D layout information during pre-training substantially improves performance over text-only models like BERT and RoBERTa, especially in low-resource form understanding scenarios. LayoutLM also shows strong cross-domain generalization on receipts and achieves state-of-the-art document classification accuracy when combining text, layout, and visual features.

On the FUNSD form understanding dataset, LayoutLM models that incorporate layout information consistently outperform text-only baselines like BERT and RoBERTa. Even a base-sized LayoutLM with relatively few pre-training pages surpasses larger text-only models, and further scaling the pre-training data yields additional gains. LayoutLM BASE with 2M document pages and 6 epochs reaches an F1 of 0.7592, substantially higher than RoBERTa BASE (0.6648) and even RoBERTa LARGE (0.7072). Increasing the number of pre-training pages from 500K to 2M improves LayoutLM BASE F1 monotonically from 0.6985 to 0.7592.

The table reports LayoutLM BASE performance on the FUNSD form understanding task when pre-training with different dataset sizes and numbers of epochs. Increasing epochs consistently raises F1 for the 500K-page setting, and using more pre-training data (1M vs 500K) also improves accuracy at the same epoch count. These trends demonstrate that both longer pre-training and larger pre-training corpora benefit downstream accuracy in a low-resource fine-tuning scenario. On 500K pre-training pages, F1 improves steadily from 0.6313 at 1 epoch to 0.6985 at 6 epochs, showing a monotonic gain with longer pre-training. Moving from 500K to 1M pre-training pages at 1 epoch raises F1 from 0.6313 to 0.6552, indicating that additional data enhances accuracy even at early stages.

Initializing the LayoutLM model with a pre-trained language model substantially improves form understanding performance on the low-resource FUNSD dataset compared to training from scratch. RoBERTa initialization consistently outperforms BERT initialization, yielding larger F1 gains for the BASE architecture than for the LARGE one. The best overall result is achieved by the LARGE model initialized with RoBERTa. Training from scratch results in the lowest F1 scores for both BASE and LARGE configurations, demonstrating the importance of pre-trained language model initialization. RoBERTa initialization outperforms BERT initialization by 2.1 F1 points for the BASE model and 1.3 points for the LARGE model. The LayoutLM LARGE model with RoBERTa initialization achieves the highest precision, recall, and F1 among all tested settings.

On the SROIE receipt understanding task, LayoutLM models that combine text and layout significantly outperform text-only BERT and RoBERTa baselines. Even a BASE-sized LayoutLM pre-trained on just 500K document pages surpasses the strongest text-only LARGE model, and performance further improves when scaling pre-training data, with LayoutLM LARGE achieving 0.9524 F1 when trained on 11M images. Adding layout information boosts F1 substantially: LayoutLM BASE with 2M pre-training pages reaches 0.9431 F1, while the best text-only model, RoBERTa LARGE, scores 0.9280. Increasing pre-training data for LayoutLM improves accuracy, culminating in 0.9524 F1 for the LARGE variant trained on 11M document images, which exceeds the top competition entry.

On the RVL-CDIP document image classification benchmark, text-only models like BERT and RoBERTa reach roughly 90% accuracy, with RoBERTa slightly ahead. Incorporating layout via LayoutLM lifts performance considerably, and increasing pre-training data from 500K to 2M pages yields steady accuracy gains, reaching 91.65% without any image features. RoBERTa models consistently outperform BERT models of comparable size, though the margins are small (e.g., 90.11% vs. 89.92% for large variants). All LayoutLM configurations surpass the best text-only model, showing that layout information is beneficial for document image classification. LayoutLM accuracy improves monotonically with more pre-training data: 91.25% (500K pages), 91.48% (1M pages), and 91.65% (2M pages) under the same 6-epoch schedule. The LayoutLM BASE model with 2M pages achieves 91.65%, outperforming even the RoBERTa LARGE model (90.11%) despite having far fewer parameters (113M vs. 355M).

LayoutLM models that incorporate layout information consistently outperform text-only baselines like BERT and RoBERTa across form understanding, receipt understanding, and document classification tasks. Scaling the amount of pre-training data and extending training epochs yield steady accuracy gains, while initializing with a pre-trained language model—especially RoBERTa—further boosts performance. Even a base-sized LayoutLM surpasses larger text-only models, confirming that layout features provide crucial signals beyond what text alone captures.