Have you ever kept a notebook of mistakes while preparing for exams? That habit is similar to the RAG chunking method introduced today.
Think of it like organizing a question bank by subject, then by chapter, then by question type, a clear hierarchy.
When it’s time to review, only the most relevant mistakes are pulled out first, these are the fine-grained chunks. If several questions hit the same chapter, it makes sense to just revisit the entire chapter instead, that’s an upward merge. Of course, review time is always limited, so once the daily budget is spent, it’s time to wrap up.
Why Chunking Quality Matters in RAG Systems
In RAG (Retrieval-Augmented Generation), the way documents are chunked directly influences the quality of both retrieval and generation. If chunks split up semantically connected information or lump unrelated ideas together, the retrieval stage can miss important context, and the generation stage may struggle to produce accurate answers.
The real issue isn’t just chunk size, it’s whether the chunks align with the document’s underlying hierarchical semantic structure. That alignment is what ultimately determines how much useful content gets recalled and how reliably the model responds.
That’s why chunking in RAG shouldn’t be treated as a one-off preprocessing step. It deserves to be optimized and evaluated systematically. Without that, there’s no clear way to quantify and improve the chunking and retrieval backbone that powers high-quality responses.
HiChunk: Hierarchical Chunking Meets Adaptive Retrieval
HiChunk blends a hierarchical chunking strategy with an adaptive retrieval-time Auto-Merge mechanism, striking a balance between semantic completeness and flexible granularity.
Learning Document Structure as a Tree
The chunking process starts by treating sentence segmentation as the base unit. Given a document split into sentences S[1:N], HiChunk uses a fine-tuned language model to predict global chunking points across multiple semantic levels GCP_{1:k}. Instead of manually identifying boundaries or applying rigid rules, the model learns both where to cut and how to structure the content into a hierarchy.
For longer documents that exceed the model’s input limit, an iterative inference strategy is used. The model processes sections of the document within a token budget L, predicts local chunking points LCP_{1:k}, and merges them into the global tree. To prevent hierarchy drift, particularly in cases where only one top-level chunk is found, residual lines from previously seen context are added to help stabilize predictions.
HiChunk is trained on datasets with explicit structural labels, including Gov-report, Qasper, and Wiki-727. To improve generalization, data augmentation is applied by randomly shuffling document sections and removing content. During training and inference, the model handles up to 8192 and 16384 tokens respectively, with each sentence capped at 100 characters.
Retrieval that Adapts to the Query
To balance semantic integrity with uniform retrieval granularity, the output of HiChunk (semantic chunks) is further divided into fixed-size chunks (referred to as HC200 in the experiments). These fixed-size chunks are sorted by similarity to the query using a shared embedding model. Retrieval proceeds step-by-step, sequentially traverses similarity-ranked chunks (a greedy traversal) using BGE-m3 and adding high-ranking chunks to the context until the token budget is hit.
What makes this stage adaptive is the Auto-Merge algorithm. Under certain conditions, multiple child nodes belonging to the same parent are merged upward into their parent node, producing larger, semantically complete chunks—if the budget allows. The merge is triggered when:
At least two child chunks of the same parent have been retrieved
The combined length of these child chunks exceeds a threshold θ∗\theta^*θ∗
Require the remaining budget (T−tkcur) ≥ len(p), then replace children with the parent and rebuild the context.
The threshold θ∗ grows with budget usage, dynamically increasing from one-third (len(p)/3) to two-thirds (2 * len(p)/3) of the parent’s length. After merging to the parent, the while-loop may continue upward to ancestors if conditions still hold.
This encourages early inclusion of fine-grained leaf chunks and late-stage promotion of larger parent chunks, improving both coverage and coherence in retrieved context.
Balancing Granularity and Completeness
This variant, called HC200, combines the semantic awareness of hierarchical chunking with the predictability of uniform chunk lengths. When paired with Auto-Merge during retrieval, the full system is referred to as HC200+AM.
For fair comparison, retrieval context is always constrained by token count rather than chunk count, typically capped at 4096 tokens. The system also supports testing under a range of budgets, from 2000 to 4000 tokens, to evaluate robustness under different resource settings.
Evaluation
Many existing RAG benchmarks fall short when it comes to evaluating chunking strategies. It’s challenging to isolate and assess chunking methods with existing benchmarks.
Most questions in these benchmarks rely on just one or two sentences of supporting evidence. When the signal is that sparse, it becomes difficult to distinguish between good and bad chunking. This stands in stark contrast to real-world, evidence-dense tasks like summarization. Even if a system retrieves the right sentence, it’s unclear whether the chunking helped or just got lucky.
HiCBench was created specifically to fill that gap.
The benchmark includes manually annotated the hierarchical structure (thus defining cut points) across multiple semantic levels, and each question-answer pair is constructed to be evidence-dense, with its required evidence intentionally distributed across one or more complete semantic chunks. Alongside each QA pair, HiCBench provides explicit evidence sentences, making it possible to evaluate not just whether an answer is correct, but whether the right information was retrieved from the right chunks.
As shown in Figure 4, for chunking accuracy, when evaluated on two in-domain datasets (Qasper and Gov-report) and the out-of-domain HiCBench, HiChunk consistently outperforms single-level semantic and LLM-based chunking methods (SC and LC) in cut-point accuracy by a large margin.
Thoughts
HiChunk addresses one of the core challenges in RAG: the tension between fine-grained retrieval granularity and preserving semantic completeness. By coupling hierarchical chunking with the Auto-Merge mechanism during retrieval, the system dynamically merges related chunks based on structural alignment and token budget, ensuring that the retrieved context is both relevant and coherent. The merge logic is driven by well-defined conditions and respects the parent-child relationships in the chunk hierarchy, making it a practical and engineering-friendly solution.
That said, in my view, Auto-Merge relies heavily on the quality of the underlying embedding model and the available token budget. Its thresholds and chunk rankings are tightly coupled with how the retriever scores relevance. When moving across corpora or switching embedding models, some recalibration may be necessary. Otherwise, the retrieval gains can vary more than expected.
Reference: HiChunk: Evaluating and Enhancing Retrieval-Augmented Generation with Hierarchical Chunking
Superb analysis! Chunking challenges human cognitive models, too.