ExtractBench is a document structured extraction benchmark dataset released by LlamaIndex in 2026, designed to provide a standardized evaluation baseline for enterprise document information extraction systems based on user-defined schemas. It supports research and model capability assessment for complex extraction tasks such as complete recovery of long lists, sparse fact localization, and dense layout field filling. The related paper is titled "[ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction]" (https://hyper.ai/papers/2607.29677).

The dataset comprises 370 real-world and synthetic documents (totaling 4,869 pages), covering eight business domains and 67 types of documents. Each record represents a test case where the input consists of the full document and a user-defined schema, while the output is a JSON object conforming to the schema specification, accompanied by source page numbers and bounding boxes for each extracted value as evidence. This benchmark labels each document across five independent dimensions—task difficulty, perceived difficulty, table structure, document length, and business domain—to facilitate attributing low scores to specific causes.

Dataset Composition:

• Divided into three subsets by length:

  • Short: 252 test cases, 615 pages; single-document limit ≤ 10 pages
  • Medium: 98 test cases, 2,438 pages; 11–50 pages per document
  • Long: 20 test cases, 1,816 pages; > 50 pages per document

• Categorized by task challenge::

  • T1 Long List Completeness: 154 documents, focusing on extracting all items within repetitive structures
  • T2 Long-Domain Fact Localization: 39 documents, focusing on accurately locating specific facts amidst massive text volumes
  • T3 Dense Document Extraction: 214 documents, focusing on field completion under conditions involving complex layouts, handwriting, or scanning interference

Data Fields:

• id: Unique identifier for the test case
• category: Dataset partition (short / medium / long)
• pdf: Relative path to the corresponding source document
• data_schema: JSON-encoded JSON Schema defining the format that outputs must validate against
• expected_output: JSON-encoded ground truth extraction results
• field_rules: JSON-encoded field-level scoring rules, including comparator type, evidence sources, and validation status
• repeated_structure: Identity key configuration used to align array records
• tags: Multi-dimensional analysis tags encompassing task challenges, perceptual conditions, table structure, length, and business domain

Citation

@misc{zhang2026extractbenchbenchmarkschemaguidedenterprise,
      title={ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction},
      author={Boyang Zhang and Adrian Lyjak and Eli Stewart and Zhaoqi Li and Simon Suo},
      year={2026},
      eprint={2607.29677},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2607.29677},
}