The Hidden Bottleneck in RAG Code Completion: How You Chunk the Code — AI Innovations and Insights 138
In a previous article, I discussed how Claude Code handles retrieval (The Secret Behind Claude Code’s Retrieval: Why Live Search Fits Better than RAG — AI Innovations and Insights 134), noting that it basically doesn’t incorporate a RAG process.
Today, let’s dive into the approach and findings related to using RAG for code completion.
Motivation
When using RAG for code completion, it’s common practice to first divide the code into chunks before retrieval. However, the question of exactly how to chunk the code hasn’t been systematically studied.
Most existing work has focused primarily on optimizing the retriever or generator, treating chunking merely as a default preprocessing step. But chunking directly influences the smallest unit of code available to the retriever. Poor chunking can degrade the quality of completions, no matter how advanced the retriever or the language model itself may be.
So here’s something worth thinking about: In retrieval-augmented code completion, how important is chunking really? Is structured chunking truly better than a simple sliding window approach? And when balancing chunk size, context length, cost, and effectiveness, what’s the best trade-off?
Retrieval-Augmented Code Completion: Key Technologies
Before jumping into the findings, let’s first unpack the key technologies involved.
Chunking: How to Divide Code for Retrieval
Chunking means breaking source code files into small, retrievable pieces that the retriever can work with later. There are four representative strategies:
Function Chunking: Extracts each top-level function or method, keeping its signature and body together when possible; oversized functions are split along direct child-statement boundaries.
Declaration Chunking: Retains class headers, field definitions, method signatures, and function-level declarations while omitting method bodies; in the Python setup, private underscore-prefixed functions are filtered out.
Sliding Window: Splits files into fixed-size line-aligned windows with configurable line overlap, without using syntax structure. This method is simple, language-agnostic, but risks splitting functions awkwardly.
cAST: Operates over all Abstract Syntax Tree (AST) node types, recursively splitting large nodes and greedily merging adjacent siblings to produce more uniform chunks.
