ExtractBench: A New Standard for Enterprise Document Extraction

What Changed

For years, the field of document AI has relied on general-purpose benchmarks that often fail to capture the nuances of enterprise-grade extraction. The release of ExtractBench marks a significant shift in how developers evaluate models tasked with schema-guided extraction. Unlike standard document visual question answering (DocVQA) tasks, which often focus on simple retrieval, ExtractBench specifically targets the enterprise requirement: given a document and a user-defined schema, an agent must faithfully extract data while providing source evidence as grounding metadata.

This benchmark is the first to simultaneously score value accuracy, record completeness at scale, grounding, and measured cost. By providing a standardized evaluation system for 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, ExtractBench moves the industry away from anecdotal performance reports toward a rigorous, reproducible standard. It forces developers to confront the reality of how models handle complex, multi-page documents—a common pain point in production-grade AI pipelines.

Technical Details

The architecture of ExtractBench is designed to solve the problem of ground-truth curation at scale. Creating a high-quality dataset for document extraction is notoriously difficult because it requires precise alignment between the extracted values and the source document. The authors of the benchmark implemented a multi-layered curation pipeline to ensure data integrity:

1. Independent-system agreement: For real-world documents, the team utilized consensus across multiple systems to validate ground truth.
2. Synthetic lists: For complex data structures, known values were injected into synthetic lists to test the limits of model recall.
3. Human verification: Forms were subjected to manual review to ensure the schema-guided extraction was accurate and properly grounded.

The evaluation metrics are equally rigorous. The benchmark reports order-insensitive value F1 for value accuracy, ensuring that the model is penalized for missing data rather than just formatting errors. Crucially, it introduces two grounding metrics for source traceability: word-level F1 and page-level F1. These metrics force models to prove where they found the information, which is a non-negotiable requirement for enterprise applications involving legal, financial, or medical data where auditability is paramount.

The dataset covers a wide spectrum of enterprise complexity, ranging from simple invoices to complex, multi-page contracts and technical manuals. By differentiating these with clear tags, the benchmark allows developers to isolate specific failure modes—such as the inability to parse tables or the tendency to hallucinate data when the schema is complex.

Developer Implications

The findings from the ExtractBench release highlight a stark divide in the current AI landscape: the trade-off between the convenience of Commercial Vision Language Models (VLMs) and the reliability of coding agents.

One of the most critical insights from the benchmark is the performance degradation of commercial VLMs on long-form documents. While these models often perform admirably on single-page documents or short snippets, they frequently truncate record lists when processing longer files. This is a silent failure mode; the model may return a partial list that looks correct, leading to downstream data integrity issues that are difficult to debug. For an enterprise application processing a 50-page invoice batch, this truncation is not just a nuisance—it is a system failure.

Conversely, coding agents—which leverage code execution to parse and extract data—consistently demonstrate higher accuracy. However, this accuracy comes at a significant cost, both in terms of latency and computational expense. The benchmark identifies that these agents are often overkill for simpler tasks, yet necessary for high-stakes extraction where completeness is the primary KPI.

The standout performer in the benchmark, LlamaExtract Agentic Plus, bridges this gap. It ranks first across all three primary metrics (value accuracy, grounding, and completeness) while maintaining a cost profile comparable to, or better than, traditional coding agents. This suggests that the future of enterprise extraction lies not in larger, more expensive models, but in specialized agentic workflows that are optimized for the specific constraints of document parsing—such as context window management, table structure recognition, and iterative verification.

For developers, the implication is clear: stop relying on general-purpose benchmarks to validate your extraction pipelines. If your application relies on document data, you must evaluate against a framework that measures grounding and completeness. The days of accepting "good enough" extraction are over, as the cost of manual remediation for hallucinated or truncated data is becoming unsustainable.

Bottom Line

ExtractBench provides the necessary infrastructure to move enterprise document extraction from a brittle, experimental phase into a robust, production-ready discipline. By exposing the limitations of commercial VLMs regarding long-form document truncation and highlighting the efficiency of specialized agentic workflows, it gives developers a clear roadmap for building reliable systems. The availability of the dataset and evaluation code on HuggingFace and GitHub is a call to action for the community to standardize how we measure the accuracy and traceability of AI-driven data extraction.

Pneumetron