Where Autoregressive OCR Starts to Break Down

Most existing OCR and visual language models (VLMs) rely heavily on autoregressive decoding, meaning they generate text tokens sequentially, one by one, from left to right.

While this approach works well for standard text generation tasks, it’s far from ideal for document OCR. Here’s why:

1. Speed Issues: Documents, especially lengthy ones filled with tables, formulas, and complex layouts, require generating many tokens. Decoding each token sequentially leads to significant latency, slowing the entire recognition process.

2. Error Propagation: Autoregressive methods are highly sensitive to early mistakes. A single recognition error can distort the context for subsequent tokens, causing a cascade of inaccuracies that build upon one another.

3. Over-Reliance on Language Priors: In Semantic Shuffle benchmark, AR models often lean heavily on linguistic cues and semantic coherence. This means they may “guess” rather than clearly perceive the actual text. When the semantic structure is disrupted or ambiguous, AR performance typically drops dramatically.

4. OCR as Inverse Rendering: Fundamentally, document OCR is better thought of as “inverse rendering.” The goal is to reconstruct structured information (like text, layouts, tables, and equations) from a two-dimensional image. The correct interpretation primarily depends on visual evidence and spatial arrangements. Forcing a strict left-to-right serialization is merely an "implementation artifact" for representation convenience, rather than a fundamental property of how documents are actually structured.

5. A Strong Fit for Diffusion: Unlike open-ended text generation (like chatting with ChatGPT), OCR is a near-deterministic task with limited semantic ambiguity. This makes OCR a strong candidate for masked diffusion, where masked tokens can be predicted in parallel conditioned on the image and partially observed sequence, producing a tunable speed–accuracy trade-off.

Given these considerations, document OCR systems would greatly benefit from decoding strategies that are parallelized, globally consistent, and strongly grounded in visual features. Rather than forcing OCR into the sequential patterns of autoregressive language generation, it’s more natural to employ methods designed specifically to exploit visual structure.

MinerU-Diffusion: From Left-to-Right OCR to Parallel Visual Decoding

MinerU-Diffusion uses diffusion-based decoding instead of the traditional autoregressive method, enabling the model to simultaneously confirm or correct multiple tokens through visual context. This approach boosts processing speed, reduces error propagation, and decreases reliance on linguistic context for guessing content.

The method can be understood through four practical components.

1. Unified Output Format

Text, layout annotations, table symbols, and formula indicators are all represented as a unified sequence of tokens.

For document parsing, the model outputs a structured sequence rather than only plain text; task-specific prompts can still produce plain text, LaTeX, or table markup.

2. Diffusion-Based Decoding Replacing Autoregression

During training, tokens are randomly masked, prompting the model to predict these masked elements based on the surrounding context and visual evidence from the document image.

At inference, the model progressively reconstructs masked positions, using already decoded context and visual features rather than generating strictly left to right. Over multiple iterative rounds, uncertain tokens are progressively revealed and corrected rather than sequentially generating each token from left to right.

3. Block-wise Diffusion

Diffusing across an entire document sequence can be slow and unstable, so sequences are divided into smaller blocks:

• Within blocks: Diffusion is parallelized, and context is considered bidirectionally.

• Between blocks: A coarse, front-to-back dependency helps preserve sequence coherence and reduce long-range drift.

• System Efficiency: The causal (front-to-back) structure across blocks naturally enables efficient KV-caching during inference, reducing memory and computation costs compared to full-attention diffusion models.

This design maintains fast parallel decoding while mitigating position drift and error accumulation common in lengthy documents.

4. Confidence-Driven Dynamic Decoding + Two-Stage Training

During inference, tokens with high confidence are confirmed first, while low-confidence tokens undergo further iterative correction. Confidence thresholds balance decoding speed and accuracy.

After multimodal initialization, training happens in two stages: initial broad-scale training provides general capabilities, followed by an uncertainty-driven refinement. The model automatically mines challenging examples (like complex tables or ambiguous boundaries) by measuring its own inference consistency, focusing its learning on the hardest cases to enhance robustness.

In short, MinerU-Diffusion treats document OCR as the inverse problem of reconstructing structured text from images, leveraging block-wise diffusion to parallelly refine tokens, and employing confidence-driven scheduling and challenging-case training to boost decoding speed, stability, and reliability.

Evaluation

Document Parsing Evaluation

MinerU-Diffusion’s capability in full-page document parsing is evaluated using OmniDocBench v1.5, measuring its performance through various metrics such as text edit distance, formula correctness (CDM), table extraction quality (TEDS), and reading order.

The results showed that MinerU-Diffusion achieved an overall score of 88.94 without using ground-truth layouts. When provided with ground-truth layouts, the score improved significantly to 93.37, coming very close to the performance of strong autoregressive OCR systems. This mainly shows that once layout errors are removed, its recognition quality is highly competitive.

Efficiency Evaluation

Efficiency was tested by adjusting the confidence thresholds, which determine how many tokens the model finalizes in a single decoding step. Lower thresholds led to faster decoding speeds, while higher thresholds improved stability.

MinerU-Diffusion achieved up to a 3.2× decoding speedup, maintaining a clear advantage in speed even at high accuracy levels.

Thoughts

At its core, MinerU-Diffusion transforms OCR decoding from sequential token-by-token generation into a visually-driven, block-wise diffusion process: tokens are refined in parallel within each block, while blocks retain a coarse front-to-back dependency.

Coupled with uncertainty-driven curriculum training, this shift represents a fundamental change at the decoding paradigm level, not merely swapping out the underlying model backbone.

But I have a concern.

Block boundaries could introduce new sources of subtle errors. While MinerU-Diffusion mitigates this by allowing tokens to causally attend to preceding blocks, they are strictly cut off from future blocks. Structures like headers, footers, table cells, or formulas spanning line breaks might still be disrupted if they fall near these boundaries. Such systemic fragmentation might not clearly surface through averaged evaluation metrics.

Reference:

• Paper: MinerU-Diffusion: Rethinking Document OCR as Inverse Rendering via Diffusion Decoding.

• Code: https://github.com/opendatalab/MinerU-Diffusion.

That story holds up right until a long industrial PDF arrives on your desk and makes one thing obvious: documents were never meant to be understood as confetti.

The Hidden Pitfalls of Naive RAG Chunking

Many RAG systems retrieve relevant chunks from documents and then ask an LLM to answer from that retrieved context.

However, there’s a catch: to make documents searchable, especially longer ones, they usually have to be split into smaller pieces. Traditional methods for splitting these documents can be overly simplistic, such as slicing by a fixed number of words or strictly relying on semantic meaning.

In practice, this approach often leads to problems. Real-world documents aren’t simply chunks of plain text, they often include:

• Multiple levels of headings and subheadings

• Tables, figures, and other layout-heavy elements

• Content spanning across multiple pages

• Scanned PDFs that require OCR processing

When documents are just “split by length,” it’s akin to randomly chopping up a textbook, headings end up isolated from their sections, paragraphs get separated from accompanying tables, and critical context can get lost altogether.

This fragmentation can make retrieval return disjointed evidence, which can reduce answer quality. The problem becomes especially pronounced when dealing with industrial documents, financial reports, legal contracts, and scanned materials.

Core Idea

To make RAG over long documents more faithful, three kinds of signals need to be modeled together:

• Visual layout (what the document looks like)

• Textual content (what the document says)

• Structural hierarchy (how the document is organized)

As shown in Figure 1, MultiDocFusion combines visual layout, text, and hierarchy; the name “Fusion” refers to that integration, while “MultiDoc” also reflects support for diverse document formats and corpus-level multi-document RAG scenarios.

An Intuitive Analogy

Traditional fixed-length methods are more like slicing a book at fixed token boundaries.

MultiDocFusion, on the other hand, acts like an intelligent reader: it dynamically reconstructs the document's structural tree (like building its own table of contents on the fly) and carefully groups content that naturally belongs together.

This makes it particularly effective for handling long, complex documents that reflect real-world complexities.

MultiDocFusion: How Does It Actually Work?

Think of it as a four-step process.

Step 1: Understanding the Document Layout (Document Parsing)

Read more

Today, the focus shifts to a new method for acquiring those skills.

Why Agents Need a Master Manual, Not a Pile of Sticky Notes

A skill is more than just a single file, it's a structured knowledge directory containing a root SKILL.md (the instruction manual) alongside auxiliary resources like executable scripts and reference files.

As agents tackle more specialized domains, the demand for highly specialized, comprehensive skill documentation grows, but human production simply cannot keep up.

There are also two major challenges:

First, asking a large language model to generate skills purely from its parametric knowledge often misses the real pain points, operational nuances, and common failure modes of a target domain. The resulting skills might look plausible on paper, but they rarely provide meaningful help in actual tasks.

Second, many existing online update methods based on trajectory data tend to fragment knowledge and overfit to local experience.

• Fragmentation: Many approaches extract a lesson from each trajectory, ending up with a heap of disconnected skills that are hard to search and apply.

• Sequential updates: Updating skills after every new trajectory is like trying to revise a guide without ever fully understanding the domain. It’s easy for a single trajectory to skew the knowledge.

Read more

In this post, let’s take a look at its latest progress.

The Key Constraint in Document Parsing

A recurring pattern has emerged: state-of-the-art models with different architectures and parameter sizes often make almost identical mistakes on challenging real-world PDF samples.

Simply switching to a larger model is unlikely to solve the problem. Instead, the bottleneck lies in a shared set of data-related challenges that limit progress across the board.

As a result, the key constraint on further advances in document parsing may no longer be model architecture, but the training data itself.

This data bottleneck has two main dimensions:

• First, insufficient coverage. Training datasets contain plenty of common page layouts, but long-tail scenarios such as complex nested tables, dense mathematical formulas, and unconventional multi-column layouts remain underrepresented.

• Second, there is an annotation quality paradox: many of the most informative samples, especially the difficult ones, are exactly the cases where automatic annotation is least reliable. Because of this, simply scaling raw data volume is insufficient: without better sampling and more reliable labels, it can amplify both distribution bias and annotation noise.

Given these challenges, an important question arises: if the architecture remains unchanged, can systematic data engineering and staged training still deliver significant gains, and can a more discriminative evaluation protocol reveal those gains more faithfully?

More provocatively: can a 1.2B-parameter model rely purely on data to outperform giants with over 200x more parameters?

MinerU2.5-Pro: Data Engine

Read more

They know what to do, just not always in the right order or at the right time.

Why Skills are Needed?

Over the past two years, discussions around Agents have often focused on bigger models, longer context windows, and more tools.

But the real challenge in production-level Agents isn’t whether the model can do something, it’s whether it can reliably complete a task the same way every time.

As Agent’s tasks grow longer, environments become more complex, and branches multiply, language models tend to drift. Steps get skipped, sequences get jumbled, tools are used inconsistently, and stopping conditions vary.

These challenges are known as procedural burden of agency.

Before diving into the introduction of Skills, let’s take a look at where they fit within an Agent.

Skills

Skills were created as an externalization response to this procedural burden.

A skill is not just a tool, and not merely a prompt file; it is an externalized form of procedural expertise. Tools answer the question “what actions can be performed,” protocols answer “how these actions are described and invoked,” and a Skill answers “how this type of task should be handled as a whole.”

A truly reusable Skill has at least three layers: the operational flow, decision heuristics, and governing constraints. In other words, it not only tells an Agent what to do first and what to do next, but also guides which path to take when there are branches, and defines which boundaries must never be crossed.

Read more

This post offers some interesting ideas.

Motivation

Hallucination remains a critical bottleneck for LLMs, undermining their reliability in real-world applications, especially in Retrieval-Augmented Generation (RAG) systems.

There are three core challenges:

• First, even with access to documents, models can misreport numbers, mix up timelines, or make reasoning jumps that ignore the evidence. In fields like finance, law, and healthcare, these mistakes can have serious consequences.

• Second, existing training signals are too coarse. Traditional supervised fine-tuning may teach a model to “sound convincing,” but not to ensure every fact is accurate. Standard reinforcement learning methods, including RLHF, usually provide a single score for the final answer. Even newer approaches like RL with Verifiable Reward (RLVR) are bottlenecked by the scarcity of expert annotations and the limited reasoning ceilings of external verifiers.

• Third, current verifiers suffer from confirmation bias due to "information leakage". When a judge sees the question, the source documents, and the model’s answer at the same time, there’s a strong tendency to justify the original response rather than independently verify it. This is a fundamental flaw in many hallucination detection methods.

MARCH: Three Agents, One Goal for Tighter Grounding and Fewer Hallucinations

MARCH is a framework that separates answer generation from answer verification and intentionally creates information asymmetry.

Read more

Why Traditional OCR Is No Longer Enough

Traditional OCR and many document parsing pipelines are still largely text-centric: they recover text well, but usually do not structurally parse information-dense graphics.

When confronted with charts, diagrams, icons, or UI components, they usually treat these elements as mere image fragments and store them as pixel blocks. As a result, a significant portion of structural and semantic information within documents is lost.

Figure 1 captures the main shift: traditional OCR keeps graphics as raster regions, while Multimodal OCR represents text and eligible visual symbols in a unified serialized format, using plain text, table markup, LaTeX, or SVG depending on the element type.

MOCR: Not Just Reading Text, but Reconstructing Graphics

MOCR, short for Multimodal OCR, aims to parse anything found in documents.

Read more

This post might offer an insightful idea.

Motivation

There are two common approaches to creating methodology diagrams and statistical plots for research papers today, and neither of them quite gets the job done.

• The first is code-based drawing, using tools like TikZ, Python-PPTX, or SVG. This approach works well for structured diagrams and precise layouts. But it starts to fall short when dealing with the kinds of visuals that show up in modern papers, things like intricate icons, custom shapes, and polished, detail-heavy designs. Flexibility becomes a real limitation.

• The second approach is to rely on image generation models. These can produce visually appealing results with very little effort. The problem is consistency. It is hard to guarantee that the output meets the standards of academic work, both in terms of factual accuracy and visual conventions. A figure might look good at first glance, but still require careful checking to ensure logical faithfulness and academic readability.

In practice, this means researchers often end up spending a significant amount of time manually fixing figures anyway, even after using these tools.

PaperBanana: Read, Plan, Stylize, and Sketch Science

PaperBanana is an agentic, reference-driven framework that tries to bridge this gap by synthesizing aesthetic guidelines from curated academic examples.

Read more

This post is interesting because it introduces a method that tackles that gap in a very practical, engineering-first way.

The Problems of Standard RAG

Standard RAG is often not reliable enough for hardcore tasks, like the WattBot 2025 Challenge, which demands ±0.1% numeric precision and exact source attribution. In practice, three issues show up again and again:

• Flat chunking breaks semantic boundaries and document structure, which makes precise source tracking much harder and weakens confidence in citation quality.

• A single query can easily miss relevant evidence when the wording does not line up, especially when different terms are used for the same idea.

• A single generation pass is inherently unstable. Both the answer and its citations can vary from run to run, and in some cases, terrified of hallucinating, the system may unnecessarily abstain (outputting a blank) despite the evidence being buried right there in the context.

KohakuRAG: From Flat Chunks to Citation-Ready Trees

At its core, KohakuRAG is a lightweight RAG framework built on a four-level hierarchical document index: document → section → paragraph → sentence.

It preserves document structure through bottom-up embedding aggregation, improves retrieval coverage with LLM-based query planning plus cross-query reranking, and stabilizes final outputs with abstention-aware ensemble voting during generation.

1. Offline hierarchical document indexing

The first step is document parsing.

Read more

The real shift is this: OCR is no longer just a reading task, but a document understanding problem.

Modern OCR Is About Understanding Documents

In the past, OCR usually meant one thing: recognizing text from an image.

Today, the scope of OCR has expanded far beyond basic character recognition. Real-world document processing rarely stops at reading text. It often involves layout analysis, table parsing, chart understanding, question answering, and extracting key information from complex pages. Documents are no longer just lines of text. They contain structure, hierarchy, and meaning that must be understood as a whole.

Because of this shift, OCR has gradually evolved from a character recognition tool into something much closer to a document intelligence system. Instead of only asking what characters are on the page, modern OCR systems are expected to understand how the page is organized and what the content actually represents.

Three Paradigms of Modern OCR

Current OCR paradigms can roughly be grouped into three categories:

• Pipeline OCR systems

• End-to-end OCR models

• General vision-language models.

These three approaches reflect different stages in the evolution of OCR, and each comes with its own set of trade-offs.

Traditional Pipeline OCR: Strong in Control, Weak in Fragmentation

The most classic design is the pipeline OCR system. A typical workflow starts with layout detection, followed by element-level recognition, and finally a rule-based step that assembles everything into the final output.

Read more

The Bottleneck of Combining RAG with Knowledge Graphs

Most GraphRAG systems today are built on top of binary knowledge graphs. Knowledge is broken down into simple triples: head entity, relation, tail entity. This representation is widely used, but its simplicity comes at a cost.

Two structural limitations stand out.

First is semantic fragmentation. Many real-world facts involve multiple entities and roles within a single relational event. When those facts are decomposed into isolated binary triples, part of the original semantic structure is lost.

Figure 1 uses an n-ary fact roughly of the form: “Bruce Seth Green, Sam Weisman, Sam Pillsbury, and Eric Laneuville directed TV 101 in English in California.”

In a binary graph, this single holistic fact must be decomposed into multiple pairwise triples, which breaks apart its original semantic unity. The connection between director, movie, and location is no longer represented as one unified event, what researchers call an n-ary relation. It becomes a set of separate statements that the system must later stitch back together.

Second is path explosion. Because meaning is scattered across multiple edges, the system has to rely on multi-hop reasoning to reconstruct the original context. This typically means deeper traversals across the graph. As the graph grows, the search space over possible paths expands quickly. Computation becomes heavier, and small errors in earlier hops can propagate forward, compounding the problem.

The Dual-Engine Architecture of HyperRAG

To exploit this expressive topology without getting lost in the noise, HyperRAG is introduced. Let's dive into how they work.

Read more

There is a question: What if the system could build a document-level view, instead of reasoning over isolated chunks alone?

The Problems of Current RAG and GraphRAG Systems

Many existing RAG and GraphRAG systems share a few structural weaknesses. They often miss the big picture, struggle with deeper reasoning, and tend to treat documents as pure text. Charts, diagrams, tables, and layout are usually processed in isolation, if at all.

There are five underlying reasons behind this.

Long Documents are Inherently Difficult to Handle

Even though modern multimodal models can process both images and text, they are still constrained by context window limits. When dealing with entire books, lengthy reports, or full lecture decks, true global understanding becomes elusive.

This limitation directly affects a model’s ability to grasp high-level concepts and the overall structure of the content. Without a holistic view, deeper reasoning suffers.

Most Existing GraphRAG Approaches Remain Text-Centric

Methods such as GraphRAG and LightRAG excel at extracting entities and relations from text, building graphs, and performing multi-hop reasoning.

However, these approaches are fundamentally unimodal. They completely overlook visual elements like charts, maps, and diagrams. In multimodal documents, these visual components often carry essential information. When they are ignored or weakly integrated, important signals are simply lost.

Chunking Fragments the Knowledge Graph

A common practice is to split documents into many smaller chunks, then extract entities and relations from each piece independently.

While this improves efficiency, it breaks cross-page and cross-section connections. Furthermore, naive, isolated extraction often fails to capture subtle cross-modal relationships (like text-to-figure links) even within the same chunk.

The resulting knowledge graph becomes fragmented, missing relationships that only emerge when viewing the document as a whole.

Manual Construction of Multimodal Knowledge Graphs Does not Scale

Some earlier work relied on hand-crafted multimodal knowledge graphs for question answering. While effective in controlled settings, this approach is expensive and difficult to scale.

Automatically building multimodal knowledge graphs and integrating them into RAG pipelines remains an open challenge that has yet to be fully addressed.

Fragmented Retrieval Spaces

Many retrieval pipelines still struggle with mixed-modality retrieval, which is why MegaRAG uses a unified embedding model for pages, entities, and relations.

MegaRAG: Build a Living Multimodal Graph for Smarter Document QA

MegaRAG takes a different route.

Instead of treating a multimodal document as scattered text snippets, it uses an off-the-shelf document parsing tool (such as MinerU, From Big Picture to Details: MinerU 2.5 Redefines Document Parsing — AI Innovations and Insights 77) to first extract text, figures, and tables, and then organizes the entire document into a single multimodal knowledge graph.

Then, when answering a question, it retrieves evidence from two places at once: the structured knowledge inside the graph and the original visual pages. Finally, it generates the answer in two stages.

The full pipeline can be broken down into four steps.

Read more

A simple question comes to mind: what if a document is meant to be understood as a hierarchical tree rather than a flat pile of chunks?

Current Problems in Semi-Structured Document Analysis

In practice, semi-structured documents show up everywhere, yet most existing techniques struggle to support natural language queries over them. These documents usually contain tables, charts, hierarchical headings, and body text. Their layouts tend to be irregular.

An analysis of one million real documents shows that over 77% include at least one table, chart, or section heading, with 61% containing tables and 40% containing charts, making the challenge both common and meaningful.

More specifically, there are three major pain points:

Read more

There is a question: what if the real problem is not reading everything, but reading the right thing at the right time?

The Main Bottleneck in RAG

On one hand, general OCR and document parsing systems are already quite advanced. Many models reach high accuracy on standard document benchmarks. On the other hand, real downstream scenarios, especially RAG setups for financial reports, technical manuals, and research papers, rarely need a full readout of an entire document. What is actually missing is the ability to extract only the portions that matter for the current question.

A critical bottleneck in many current visual RAG systems is rigid page-level chunking and retrieval. This level of granularity is too coarse. Each page carries a large amount of irrelevant material such as headers, footers, decorative elements, and unrelated paragraphs. All of it is delivered to the generation model. That noise dilutes attention, and the limited visual token budget forces the system to compress high resolution pages. Important details, including tables, small text, rotated sections, and formulas, are more likely to be lost. The risk of hallucination rises as a result. Recent studies also show that multimodal models struggle to pinpoint evidence precisely in these scenarios and are easily distracted by irrelevant visual signals.

A better approach would let the model read a document the same way a person does. First look at the layout, then locate the relevant region, zoom or rotate if needed, and finally extract only the evidence that answers the question rather than feeding in the entire page.

Overview of AgenticOCR

Figure 1 uses a concrete example to show how AgenticOCR helps a visual RAG system answer a question.

Read more

So there is a very human question: what if models could navigate a document more like we do?

Limitations of Existing Agentic Search Systems

Existing agentic search systems can actively look through long documents, but they still treat everything as flat chunks. They rarely make use of headings, hierarchy or reading order. As a result, they miss information, repeat searches, and suffer from severe context fragmentation.

When people read long documents, the usual approach is to locate the right section first and read the surrounding content contiguously, instead of guessing keywords again and again. Many current agentic search systems still lack this ability.

At the same time, modern OCR and document parsing tools already do a decent job recovering headings, lists, and reading order into structured formats like Markdown. These structural elements are available today and should not be ignored.

This led to an idea: give the model explicit access to document structure so it can reason in a more human way, starting with locating the right part of the document and then reading through it.

DeepRead: Not More Rounds of Search, But Setting the Document Upright First

The idea behind DeepRead is straightforward: restore the document into a form that looks closer to how a person would read it.

Read more

That gap between “the file exists” and “the information is usable” is exactly why Visual Document Retrieval feels so important right now.

Why Visual Document Retrieval (VDR) Is Essential in the MLLM Era

Visual Document Retrieval (VDR) focuses on visually rich documents rather than natural-image retrieval. It focuses on PDFs, academic papers, reports, invoices, and other structured pages. These visual documents differ from natural images in three fundamental ways.

• First, they carry far denser information. Text, layout, and charts work together to convey meaning.

• Second, the retrieval granularity is much finer. People often need a specific fact inside a table, a particular sentence in a paragraph, or information tied to a location on the page.

• Third, the goal leans toward precise information access, question answering, and evidence reasoning rather than concept matching like deciding whether two images look similar.

Because visual documents encode meaning through text, layout, and graphics together, OCR-plus-text retrieval often misses important signals. Modern VDR increasingly benefits from models that preserve visual structure directly from document images.

Three Paradigms of Visual Document Retrieval

Read more

Is there a way to keep memory useful without turning your context window into a landfill?

Why RAG Doesn’t Fit Well in Agent Use Cases

Classic RAG is built for large, heterogeneous corpora where retrieved passages are relatively diverse, and a mainstream failure mode is pulling in irrelevant or redundant content.

In contrast, agent memory is a bounded, coherent dialogue stream with many near-duplicate spans, where aggressive compression can delete temporally linked prerequisites and fragment evidence chains.

So using RAG in agent scenarios often leads to:

• In highly correlated dialogue streams, fixed top-k similarity retrieval can collapse into near-duplicate chunks, repeatedly returning redundant context.

• Post-retrieval pruning can delete temporally linked prerequisites and fragment evidence chains, often hurting multi-hop and temporal QA.

xMemory: Build a Hierarchical Memory That Cuts Tokens Without Losing the Plot

So here comes an intuition: Instead of searching your entire inbox for the top-20 emails that match a keyword, you first group emails into threads, extract the durable facts each thread contains, then pull a few threads and only open individual emails when you’re still unsure.

Read more

Is there a more elegant way to preserve and leverage the document hierarchy?

Motivation

Most existing approaches to multimodal document question answering, especially chunk-based RAG, lose a critical piece of the puzzle when they slice documents into chunks and run vector retrieval. The original structure of the document, which was designed for human understanding, gets stripped away. This loss directly impacts both retrieval accuracy and reasoning quality.

There are three core challenges:

1. Multimodal documents are inherently complex and heterogeneous. Text, images, tables, and charts all coexist, and their relationships often hold the key to deeper reasoning. Modeling these connections, however, is far from straightforward.

2. Conventional chunk-based retrieval flattens the document. Hierarchical and contextual cues get lost, which breaks causal chains, causes inconsistencies in parallel reasoning, and fails to resolve references like “Table X” or “Page X”.

3. Recent systems (e.g., MDocAgent) leverage LLMs/LVLMs together with whole-page layout to refine retrieval results; more generally, LLMs can also enhance retrieval via query rewriting or re-ranking. However, these approaches still do not explicitly model the document’s native hierarchical structure — the very thing humans rely on to locate, comprehend, and reason through information.

DMAP: Two Agents and One Structural Map

The DMAP workflow can be summed up in two simple steps:

• First, turn the document into a structural map: M = SSUA(D)

• Then, answer questions using that map: A = RRA(q | M)

At the center of this process are two key agents: SSUA and RRA.

Read more

This post is about building systems that make that moment far less common.

Why Scientific Diagrams Are So Much Harder Than Natural Images

If you ask a model to draw a cat, and the worst outcome is that it looks a bit off.

But if you ask it to draw a circuit, and a missing capacitor or a slightly wrong angle can lead to a complete breakdown in logic.

This is what’s known as visual–logic divergence: the image might appear fine at first glance, but it’s scientifically incorrect. That makes it unreliable for any kind of downstream reasoning.

A key reason is that scientific diagrams are governed by strict geometric, physical, or relational constraints, which text-to-image models often fail to follow precisely.

Two Paths: One Optimizes for Expressiveness, the Other for Precision

Scientific image generation can be divided into two main approaches:

• Pixel-based methods translate text directly into pixels. They tend to produce richer visuals, but often struggle to reliably meet structural constraints.

  • Next-generation closed-source models (like Nanobanana-Pro and GPT-Image-1.5) set a high bar, yet still struggle with dense data errors compared to code-based approaches;

  • Open-source models (like HunyuanImage-3.0 and Qwen-Image) can look acceptable in some cases, but frequent text blur and semantic/structural misalignment, with substantially lower structural-correctness metrics than proprietary models.

• Code-driven or programmatic methods, on the other hand, first generate executable code, then render the image deterministically. This route offers much tighter control over structure, though it may come at the cost of visual richness. These generate diagrams by explicitly defining elements through code, using Python to specify coordinates, angles, and layout before rendering. The upside is precision. The downside is that the visuals can feel rigid or lack stylistic variety.

While there is a fundamental precision–expressiveness trade-off, this study proposes a 'Spiral Co-evolution' hypothesis, which is mutual: code-based reasoning can transfer to pixel-based models, while pixel-based diversity enriches training data and feeds back into code-based reasoning and LMM learning.

Figure 2 illustrates this nicely. For tasks that demand strict accuracy, like plotting function curves, code-based approaches are far more stable. But for visually complex scenes where detailed rendering matters, pixel-based models often produce more compelling results.

ImgCoder: A Three-Step Workflow That Puts Logic First

Read more

This post is about a neat idea: deciding when to retrieve evidence by looking outside the model’s self-confidence to objective pre-training corpus statistics.

Motivation

The trigger signals used by most dynamic RAG systems to decide whether to retrieve are often unreliable. That’s because they typically rely on internal uncertainty measures like logits, probabilities, entropy, or attention weights. But LLMs aren’t well-calibrated. They can sound confident even when they’re completely wrong, which means these internal signals don’t always line up with actual correctness.

As shown in Figure 1, the DRAGIN example highlights this failure. A model might flag a token from the question (part of a movie title), like 'Il', as highly uncertain, while treating a completely hallucinated entity (like the wrong director name) as low uncertainty, giving false confidence to something that isn’t real.

QuCo-RAG: Replacing “Confidence” with “Corpus-Based Evidence”

QuCo-RAG (Quantifying Uncertainty via Pre-training Corpus for Dynamic RAG) introduces a method for grounding answers in hard evidence: it draws statistical signals from the pretraining corpus to replace subjective confidence with measurable corpus statistics (entity frequency and entity-pair co-occurrence) from the pre-training corpus. It enables Infini-gram to provide millisecond-level hallucination detection and dynamic retrieval triggering.

As shown in Figure 2, QuCo-RAG introduces a two-stage detection strategy to decide when retrieval should be triggered:

• Before generation begins, it checks if the input question contains rare entities, specifically, when the average frequency of the entities is fewer than τ_entity (default 1000, it reports stability across a wide range (1000–10^7). This is done using Infini-gram (a suffix array-based engine that supports millisecond-latency queries over trillion-token corpora) to look up entity frequency.

• During generation, it monitors whether the co-occurrence of the entity pair being generated is supported by the corpus. If two entities have never co-occurred within the defined window (approximately 1000 tokens in experiments, with τ_cooc = 1 by default), QuCo-RAG (using Infini-gram) detects this zero co-occurrence in real time. This helps catch hallucinated facts as they emerge.

In short: detect rare entities ahead of time, block hallucinations in real time.

For implementation, what QuCo-RAG does is as follows:

Read more