Taming Chaotic Layouts: SFT + Layout-Centric RL for Document Understanding — AI Innovations and Insights 75

Complex layouts and reading order have always been among the trickiest parts of document understanding.

I’ve explored a few heuristic methods in the past, such as in “Advanced RAG 02: Unveiling PDF Parsing”:

• https://aiexpjourney.substack.com/p/advanced-rag-02-unveiling-pdf-parsing-b84ae866344e

• https://pub.towardsai.net/advanced-rag-02-unveiling-pdf-parsing-b84ae866344e

In this article, I’ll introduce an innovative approach: it treats reading order as a first-class training objective.

End-to-end LVLMs often get lost when faced with complex layouts like multi-column newspapers or posters. They struggle to preserve the natural reading order and overall structure, which limits their ability to interpret these documents accurately. At the heart of the problem is the way most models are trained: they focus on token-level alignment, without any explicit signals about paragraph grouping, spatial zones, or reading sequence.

Pain Point #1: Standard supervised fine-tuning (SFT) with cross-entropy loss doesn’t really care if a model scrambles the reading order of paragraphs or slightly misplaces bounding boxes. There’s no direct penalty for those kinds of global structure errors. As a result, models can look “right” at the token level while being completely off in terms of overall reading logic.

Pain Point #2: On the visual side, many LVLMs still rely on coarse-grained image-text alignment or fixed-resolution inputs. That’s fine for big, clean text—but it falls apart when the document is full of tiny fonts, dense layouts, or intricate structural cues that actually matter.