Why AI PDF Tools Struggle With Complex Layouts

A PDF is not a document in the way a Word file is a document. It is a page-description format: a set of drawing instructions telling a viewer exactly where to place each character, line, and image on a fixed canvas, with no requirement to record what any of it means structurally. That design choice, inherited from Adobe's PostScript printer language when PDF launched in 1993, is the root cause of nearly every failure mode AI extraction tools run into today, from scrambled reading order to misread tables.

PDF stores where things are drawn, not what they are

Open a PDF's underlying content stream and you will not find the words "paragraph," "column," or "table." You will find operators that say, in effect, "draw the glyph 'T' at coordinate (120, 450) using this font at this size." PDF descends directly from PostScript, Adobe's page-description language built for printers, and printers only ever needed to know where ink goes, not what the ink represents (Wikipedia: PDF). The format was formalized by Adobe's "Camelot" team under co-founder John Warnock and first shown publicly in January 1993; it later became an open ISO standard, ISO 32000-1 in 2008 and ISO 32000-2 in 2017, but neither revision added a mandatory reading-order model to a plain PDF page (ISO 32000 history).

This matters because reading order, the sequence in which text should be interpreted, is a property PDF never requires. A tool that extracts "raw text" from a PDF is usually just walking through the content stream in the order the original layout software wrote it, which is not necessarily the order a human eye follows across the page.

Reading order — the sequence in which the words on a page are meant to be interpreted by a reader, as distinct from the order in which they happen to be stored or drawn. In a Word document, reading order is implicit in the document's structural model (paragraphs, sections, list items). In a PDF, it has to be reconstructed, because the format was never required to store it.

Two-column academic papers are the textbook example. If a PDF-generation tool writes all of column one's text operators first and then all of column two's, extraction is trivial. But many PDF producers interleave text as they lay out the page, writing a line from column one, then a line from column two, alternating down the page in whatever order their internal rendering pipeline happened to process objects. An extraction tool with no layout awareness just concatenates text in stream order, which can braid two unrelated columns into one unreadable paragraph, sentence fragments from the left column bleeding into sentence fragments from the right. This happens before any AI model is involved. It is a property of the file format itself, and it explains why even well-funded PDF tools occasionally produce output that looks like it was shuffled.

Multi-column academic papers are a worst case, not an edge case

Scientific papers, financial filings, and newspapers share a layout pattern that is unusually hostile to naive extraction: dense multi-column text, frequently interrupted by figures, tables, footnotes, and running headers that break the visual flow a reader takes for granted. A human reader unconsciously skips the page header, reads column one top to bottom, jumps to column two, and treats a mid-page figure as a visual island rather than inline text. None of that logic is free; a tool has to infer it.

Academic PDFs add a further complication: figures and tables are frequently anchored to a specific position on the page by the typesetting system (LaTeX is a common offender here), which means a figure's caption can be spatially separated from the figure itself by a full column of unrelated body text. An extraction pipeline that walks the page by simple top-to-bottom, left-to-right coordinate order will often attach the wrong caption to the wrong figure, or interleave a caption's text into the middle of an unrelated paragraph that happens to sit at a similar vertical position in the adjacent column.

Adobe's own PDF/UA (Universal Accessibility) standard, an ISO specification adopted to make PDFs usable with screen readers, exists specifically because standard PDFs lack a reliable structural tag tree. That means the same reading-order and table-parsing failures that trip up AI extraction tools also make plain PDFs a long-standing accessibility problem for blind and low-vision readers using assistive technology, a decades-old issue that predates modern AI document tools entirely.

Layout-analysis systems typically approach this with a two-stage strategy: first, detect the geometric regions on the page (this block is a column, this box is a figure, this strip is a header), then determine the reading sequence between those regions using rules or a learned model. Getting the first stage wrong, misclassifying a wide table as two narrow text columns, for instance, cascades directly into a wrong answer at the second stage, because the model is now sequencing regions that were never real page structures to begin with.

Tables are a visual illusion, not stored data

What looks like a clean table in a rendered PDF is, in the underlying file, usually nothing more than text and thin rule lines positioned to resemble a grid. There is no equivalent of a spreadsheet's cell reference or an HTML <table> element's row and column tags. A tool has to infer "this text belongs to row three, column two" purely from geometry: how close is this text block to that ruled line, is there consistent whitespace suggesting a column boundary, do several rows of text share the same left-edge alignment.

This inference problem gets measurably harder with real-world table variety. Merged cells break the assumption of a uniform grid. Tables that span a page break force a tool to recognize that the continuation on page two is logically the same table as the one that ended on page one, with no explicit marker saying so. Nested tables, a table inside a table cell, break the simple row-and-column model entirely. Financial reports are particularly prone to all three at once: multi-row headers, merged total rows, and tables that continue across page boundaries.

Researchers have built dedicated benchmarks because this problem is hard enough to need one. Microsoft Research's PubTables-1M dataset, released to standardize evaluation of table detection and structure recognition, and the ICDAR table-recognition competitions (cTDaR) both exist because table extraction accuracy varies enormously depending on table complexity and which benchmark it's measured against (PubTables-1M, arXiv 2110.00061; ICDAR 2019 cTDaR competition). One widely cited transformer-based table-recognition model (TATR), evaluated on the ICDAR-2013 benchmark, reached roughly 65 percent exact-match structure accuracy when trained only on PubTables-1M data, and around 81 percent when trained on a combined dataset after the benchmark's own annotation inconsistencies were cleaned up (Aligning Benchmark Datasets for Table Structure Recognition, arXiv 2303.00716). Those numbers are specific to one model on one benchmark and should not be read as a universal accuracy figure for table extraction generally, but they illustrate a real point: even purpose-built table-structure models get a meaningful share of tables wrong on a well-controlled academic benchmark, and real-world documents are typically messier than benchmark documents.

Extraction approach	How it works	Strong on	Weak on
Rule-based text extraction	Reads the PDF content stream directly, pulling text by coordinate order or simple heuristics (whitespace gaps, font changes)	Single-column text, simple invoices, born-digital PDFs with clean structure	Multi-column layouts, any scanned document, complex or merged tables
OCR (optical character recognition)	Treats the page as an image, uses pattern recognition to convert pixels back into characters	Scanned documents, faxes, photographs of text	Reading order (OCR alone gives characters, not structure), handwriting, low-resolution scans
Layout-aware ML models (e.g., LayoutLM family)	A transformer trained jointly on text content and its 2D spatial position on the page	Forms, receipts, semi-structured business documents with recognizable visual patterns	Highly irregular or creative layouts outside the training distribution, novel document types
Computer-vision table detectors (e.g., table-transformer approaches)	Detects table regions and cell boundaries as a visual object-detection problem	Ruled tables with clear grid lines	Borderless tables, merged cells, tables spanning a page break

No single approach wins across every document type, which is why production document-AI pipelines typically combine two or more of these rather than relying on one.

Scanned documents add a second, compounding layer of uncertainty

A scanned PDF has no text layer at all; every page is functionally a photograph. Any extraction has to start with optical character recognition (OCR), the process of converting the shapes of pixels back into character codes, and OCR carries its own, separate error rate before layout analysis even begins.

OCR accuracy degrades predictably under specific conditions: low scan resolution, unusual or decorative fonts, handwriting, watermarks or stamps overlapping the text, and skewed or rotated pages all increase the character-level error rate. Speech recognition systems show a structurally similar pattern, where a model's accuracy depends heavily on how well its training data resembles the input it's given; the same underlying idea, that a system trained on one data distribution degrades on inputs that fall outside it, applies just as much to OCR systems as it does to speech transcription tools.

The compounding problem is specific to document AI and worth stating precisely: a layout-analysis or table-structure model has to work on top of OCR's output, and it has no independent way to know which characters OCR got wrong. If OCR misreads a single decimal point, mistaking a "1" for a "7," or drops a negative sign, that error doesn't just corrupt one character in the final output. If the misread character shifts where a downstream model believes a column boundary sits, it can shift the interpreted structure of an entire row, reassigning several correctly-read numbers to the wrong columns even though those numbers themselves were extracted perfectly. The error moves from a character-level mistake to a structural mistake, and structural mistakes are far harder to spot on inspection because the output still looks plausible.

Where layout-aware models actually help, and where they stop helping

Layout-aware document-understanding transformers were built specifically to close this gap. Microsoft Research's LayoutLM family, and its successor LayoutLMv3, are trained on text content and the text's 2D bounding-box position on the page simultaneously, rather than treating extraction as a purely linear text-stream problem. LayoutLMv3 combines text embeddings (derived from OCR output), layout embeddings encoding each token's position on the page, and image embeddings from the page itself, trained together so the model learns cross-modal relationships between what a piece of text says and where it visually sits (LayoutLMv3, Microsoft Research / arXiv 2204.08387).

The practical benefit is that a model like this can learn, for example, that a short line of text in the top-right corner of an invoice is usually a date and a line near the bottom labeled with a dollar sign is usually a total, regardless of the exact wording or font, because it has seen thousands of documents where that spatial pattern held. That's a meaningful improvement over pure text-stream extraction, and it's why layout-aware models perform well on the semi-structured business documents they're typically evaluated on: invoices, receipts, and forms, which share recognizable visual conventions.

Why layout awareness doesn't fully solve irregular documents

Layout-aware models learn statistical patterns from their training documents, so they generalize well to documents that resemble what they were trained on and less well to genuinely novel layouts. A hand-designed infographic, an unusually formatted legal contract, or a creatively laid-out marketing PDF can fall far enough outside a model's training distribution that its learned spatial priors stop applying, the same distribution-dependence problem that affects most trained models rather than a flaw unique to document AI.

A concrete failure case: the misplaced footnote

Consider a corporate financial report where a footnote reference, a small superscript number, sits at the end of a data cell inside a table that spans a page break. Footnote markers are frequently added to a page as a separate text layer during document assembly, meaning the footnote's position in the PDF's underlying text stream can be entirely disconnected from its visual position on the page. An extraction tool reading purely by stream order, or even a layout-aware model working from an imperfect region segmentation, can attach that footnote's reference number to the wrong data cell.

The output of that error is genuinely dangerous in a specific way: it looks completely fine. The extracted number is a real number that was correctly read from the page; it is simply now paired with the wrong row or the wrong footnote's explanatory text. Nothing about the output format signals that anything went wrong, which is exactly why this class of error survives casual review and only surfaces when someone cross-checks the extracted figure against the source document line by line.

This is the reason serious document-processing pipelines in financial services and legal work treat complex-layout extraction differently from simple text extraction. A common pattern is to pair automated extraction with a validation step, either a second, independent extraction method used to cross-check the first, or a mandatory human review before extracted data feeds into any downstream system, precisely because a misattributed footnote or a shifted table value carries real financial or legal consequences (PDF/UA and structural tagging background, ISO / Adobe). The underlying lesson is not that layout-aware models are unreliable. It's that reading order and cell attribution are genuinely ambiguous problems baked into the PDF format itself, and no current extraction approach removes that ambiguity for every document structure that exists in the wild.

What people get wrong about "just use AI to read the PDF"

The most common misconception is treating PDF extraction as a solved problem because large language models are visibly good at summarizing and answering questions about text. Those are separate capabilities. A model can write a fluent, confident-sounding summary of extracted content and still have been fed structurally wrong input, a footnote attached to the wrong row, a caption swapped with an unrelated paragraph, because the extraction step happened before the language model ever saw the text. The same mechanism that lets language models hallucinate confident, plausible-sounding but false claims means a model summarizing bad extraction output has no internal way to flag that the underlying data was already scrambled; it will describe the wrong numbers just as fluently as it would describe the right ones.

A second common misconception is assuming OCR quality and layout understanding are the same problem, or that solving one solves the other. They are separate, sequential steps that compound: a scanned document needs OCR before any layout analysis is possible, and layout models have no independent mechanism to detect or correct OCR's character-level mistakes. Improving one does not automatically fix errors introduced by the other.

A third misconception is assuming that because a tool uses a modern transformer architecture, similar in spirit to the self-attention mechanisms that let transformer models weigh relationships between words in a sentence, it must therefore understand document structure the way a person does. Attention over text tokens and attention over a page's 2D spatial layout are related mechanisms but solve different problems; a language model's fluency with text says nothing about whether the layout-parsing step upstream of it produced correctly ordered, correctly attributed input in the first place.

The takeaway

Complex layouts are hard for AI PDF tools because the format itself was designed to encode appearance, not structure. Multi-column reading order, ambiguous table boundaries, and OCR errors that compound into structural mistakes are problems rooted in how PDF works at the file level, not just gaps in current AI models. Layout-aware models like the LayoutLM family measurably reduce these errors on documents that resemble their training data, but published benchmark accuracy for even purpose-built table-recognition models sits well short of 100 percent on standardized tests, and no current approach eliminates the ambiguity for every document type. Any high-stakes extraction from a complex PDF, financial, legal, or academic, still deserves a human spot-check against the source document rather than blind trust in the output.

---

Read next:

• How transformers pay attention to words
• Why AI transcription tools make different mistakes
• Why AI hallucinates facts

FAQ

Q: Why does copying text from a PDF sometimes produce scrambled word order?

PDFs store character positions on a page as drawing instructions, not a required reading order, so multi-column layouts can have their underlying text stream interleaved between columns. Extracting text by simple stream order then produces output where sentence fragments from unrelated columns run together.

Q: Why do AI tools struggle with tables in PDFs more than tables in Excel or HTML?

A PDF table is usually just text and lines positioned to visually resemble a grid, with no explicit row or column data attached, unlike a spreadsheet cell reference or an HTML table tag. Tools must infer structure from spacing, alignment, and ruled lines, which breaks down with merged cells, nested tables, or tables spanning a page break.

Q: Does OCR quality affect layout understanding, or are they separate problems?

They are separate but compounding problems. A scanned PDF requires OCR before any layout analysis can happen, and a character-level OCR error, like a misread digit, can shift how a downstream table-structure model interprets an entire row, turning one small mistake into a larger structural one.

Q: What is LayoutLM and how is it different from regular text extraction?

LayoutLM, and its successor LayoutLMv3, are Microsoft Research models trained on text content, the text's 2D position on the page, and the page image together, letting the model learn visual patterns like where dates or totals typically appear. That's a meaningful improvement over treating a document as a flat stream of words, though it does not fully generalize to layouts far outside its training data.

Q: Can I trust AI-extracted data from a complex PDF without checking it?

Not for anything with real financial, legal, or academic consequences. Because extraction errors like a misattributed footnote or a shifted table value can produce output that still looks entirely plausible, serious document-processing pipelines pair automated extraction with a second independent check or mandatory human review before the data is used downstream.