{"slug":"rag-retrieval-scaling","title":"Why does RAG accuracy degrade as the knowledge base grows, and what fixes it?","question":"Why does RAG accuracy degrade as the knowledge base grows, and what fixes it?","summary":"Naive top-k vector retrieval treats every chunk as an independent nearest-neighbor hit, so as a knowledge base grows, questions needing several chunks combined get harder to answer in one shot. The fix is structural (graphs, agentic re-retrieval, precomputed task views), not a bigger k.","status":"active","cluster":"retrieval","cluster_label":"Retrieval and grounding","updated":"2026-10-06","latest_evidence_added":"2026-08-05","audience":"strong-software-engineer","math_depth":"intuition","sections":[{"heading":"Builder consequence","html":"

A RAG demo on a few hundred documents can look solid, then get worse every month as the knowledge base grows, because retrieval was never the part that scales. If production misses cluster on questions needing two or more facts stitched together, that is not a prompting or model-size problem. Flat top-k similarity search has hit its structural ceiling, and the fix belongs before generation.

"},{"heading":"Short answer","html":"

Naive RAG makes one independent top-k similarity lookup per query. As the corpus grows, the odds that every fact a multi-document question needs lands in that one window shrink. Three structural fixes attack this from different sides:

\n\n

Each moves structuring work ahead of the similarity search.

"},{"heading":"Builder model","html":"

Flat vector RAG answers "what single chunk is most similar to this question," not "what set of chunks together answers it." A one-chunk question stays reliable at any corpus size. A question needing two or three chunks requires all of them to land in the same top-k window, which gets harder as more distractor chunks compete for the k slots.

\n

The three fixes are different levers on that problem:

\n\n

They combine; you do not have to pick exactly one.

"},{"heading":"Mechanism","html":"

Baseline. Standard RAG (Lewis et al., 2020) embeds the query, retrieves the k nearest passages, and conditions generation on them. The generator sees only what similarity search surfaced and cannot ask for more.

\n

GraphRAG (Edge et al., 2024) targets corpus-spanning questions, such as "what are the major themes across these documents," where no single chunk is the answer. It builds an entity and relationship graph offline, clusters it into communities, and precomputes a summary per community. A query combines relevant community summaries instead of hoping one vector hit contains everything.

\n

Agentic RAG turns retrieval into a loop: plan what is needed, retrieve, check whether the context covers the question, and retrieve again (possibly reformulated) if not. A pass that misses one of three needed facts gets a second chance instead of silently generating from incomplete context.

\n

Task-aware compression (the AWS "Beyond RAG" pattern) pre-compresses the knowledge base into task-specific representations at multiple fidelity tiers, caches them, and routes each query to the tier with enough context for its task shape. It trades index-build cost and staleness risk for lower per-query cost and a ceiling that does not depend on top-k luck.

"},{"heading":"Math intuition","html":"

Take a question needing m facts, each in a different chunk, a corpus of n chunks, and a retrieval budget k. Each needed chunk's odds of landing in the top-k window shrink as n grows relative to k. If those odds are roughly independent, the chance that *all* m land together is roughly their product.

\n\n

So RAG does not decline uniformly with corpus size. Multi-hop, cross-document questions fail first, which is exactly what each structural fix targets.

"},{"heading":"How to apply","html":""},{"heading":"Failure modes","html":""},{"heading":"Related","html":"

See /topic/vector-kb for the retrieval baseline (vector vs. graph indexes) and /topic/grounding for why an answer is only as trustworthy as what was retrieved.

"}],"evidence":[{"id":"lewis-2020-rag","kind":"theory-paper","tier":"theory/paper-backed","title":"Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks","note":"Introduces the RAG mechanism this space builds on: a dense retriever fetches the top-k passages by vector similarity, and the generator conditions on them. Retrieval and generation are separate stages, so the generator can only use what similarity search surfaced.","added":"2026-08-05","url":"https://arxiv.org/abs/2005.11401"},{"id":"edge-2024-graphrag","kind":"theory-paper","tier":"theory/paper-backed","title":"From Local to Global: A Graph RAG Approach to Query-Focused Summarization","note":"Shows flat top-k retrieval fails on questions that connect facts across many documents, such as query-focused summarization over a whole corpus. Builds an entity/relationship graph and community summaries offline so a query traverses structure instead of depending on one chunk's vector coming back as the top hit.","added":"2026-08-05","url":"https://arxiv.org/abs/2404.16130"},{"id":"story-355c8cf2c3a4e36a-agentic-rag-survey","kind":"story","tier":"source story","title":"Towards Trustworthy and Cost-Efficient Data Integration: From Naïve RAG to Agentic RAG","note":"Survey tracing naive RAG to GraphRAG/KG-RAG to Agentic RAG from a data-integration angle. In Agentic RAG a multi-agent loop plans, retrieves, refines, and re-reasons instead of taking one retrieval pass as final, in response to accuracy and cost problems in enterprise deployments.","added":"2026-08-05","sid":"355c8cf2c3a4e36a"},{"id":"story-46be0149e39dc713-aws-takc","kind":"story","tier":"source story","title":"Beyond RAG: Task-aware knowledge compression for enterprise AI on AWS","note":"Vendor post reporting that flat RAG hits a ceiling on analytical tasks spanning hundreds of documents. Describes pre-compressing a knowledge base into task-specific representations at multiple fidelity tiers and routing each query to the tier with enough context, moving structuring work from query time to index time. No independent benchmark.","added":"2026-08-05","sid":"46be0149e39dc713"},{"id":"rag-scaling-editorial-synthesis","kind":"editorial-inference","tier":"editorial inference","title":"One fix does not cover every degradation mode","note":"Editorial synthesis: graph structure, an agentic retrieve-and-check loop, and task-aware pre-compression are not interchangeable. They target different symptoms (missing cross-document links, under-coverage on complex questions, per-query cost), and a system can need more than one. No source here claims one fix supersedes the others.","added":"2026-08-05"}],"related_topics":[{"slug":"vector-kb","title":"External knowledge base: vector and graph retrieval"},{"slug":"grounding","title":"An agent's answer is only as good as what it retrieved — and whether it can prove it"}],"related_playbook_cards":[],"related_storylines":[]}