Every RAG system you have used retrieves from a flat pile of text chunks. That works fine when the question is about a single document or a specific fact. But what happens when a user asks a question that requires reasoning across organizational hierarchies, product taxonomies, or regulatory structures? Standard RAG systems flatten all of that structure into a searchable blob and lose the relationships that make the data useful in the first place. A new paper on arXiv by Pranav Yadav introduces HG-RAG (Hierarchy-Guided Retrieval-Augmented Generation), a framework that replaces flat retrieval with graph-traversal over hierarchical knowledge graphs. The results are striking: on queries involving multi-level reasoning, HG-RAG outperforms standard dense passage retrieval by 30 to 50 percentage points depending on the task.
What Problem Does HG-RAG Solve?
The core insight is deceptively simple. Standard RAG systems treat all documents as equal pieces of text in a flat vector store. When a user asks something like "What products fall under the electronics category and what are their safety certifications?", the system needs to understand that electronics is a parent category of subcategories like laptops and smartphones, and that safety certifications apply at the product level, not the category level. A flat retrieval system might return chunks about electronics, chunks about laptops, and chunks about safety certifications, but it has no mechanism to connect them correctly. The LLM receiving that flat context has to guess the relationships, and guessing leads to hallucination.
The paper identifies four distinct query types that flat retrieval handles poorly: local fact queries that need entity-level precision, hierarchical queries that require understanding parent-child relationships, neighborhood queries that need lateral connections between sibling entities, and multi-hop queries that traverse multiple levels of a hierarchy. Enterprise use cases touch all four of these patterns. Product catalogs, org charts, medical ontologies, legal document structures, supply chain data, and compliance frameworks all encode meaning in their structure, not just their text. When a RAG system flattens that structure, it discards information that is essential for correct answers.
How HG-RAG Works: Graph Traversal Over Flat Retrieval
HG-RAG replaces the flat retrieval pipeline with a four-stage graph-traversal approach. Stage one resolves a named entity anchor from the user query using entity linking. If a user asks about "electronics safety certifications," the system identifies "electronics" as the anchor entity in the knowledge hierarchy. Stage two identifies the relevant subgraph around that entity, establishing boundaries for traversal. Stage three performs the actual graph traversal, collecting context across three dimensions: upward through parent nodes for broader context, laterally through sibling and relational neighbor nodes for context at the same hierarchical level, and downward through child nodes when the query requires specific details. Stage four delivers the structured context to the LLM for generation.
The framework is evaluated across three world scales (18 nodes, 200 nodes, and 800 nodes) to test scalability. The small scale represents a simple department org chart. The medium scale models a product catalog with categories and subcategories. The large scale approximates a regulatory compliance document hierarchy. At each scale, HG-RAG is benchmarked against a standard dense passage retrieval baseline using the same underlying language model.
Benchmark Results: Where HG-RAG Wins
The results show consistent advantages for HG-RAG across all structured query types. On hierarchical queries, which require understanding parent-child relationships, HG-RAG outperformed the flat baseline by roughly 45 percentage points at the medium scale and by a wider margin at the large scale. On neighborhood queries, which require lateral connections between sibling entities, the advantage was roughly 35 percentage points. On multi-hop queries that traverse multiple hierarchy levels, HG-RAG maintained strong performance while the flat baseline degraded significantly as query complexity increased.
Critically, HG-RAG did not sacrifice performance on local fact queries, where a flat baseline might be expected to perform well. The graph-traversal approach returned comparable results for simple single-entity queries, meaning there is no trade-off: you get better structured reasoning without losing fact-level accuracy. The paper also reports reduced hallucination rates, since the LLM receives pre-structured context rather than having to infer relationships from a flat chunk list.
The limitations are worth noting. The framework assumes that a hierarchical knowledge graph exists or can be constructed for the domain. Building and maintaining that graph is a non-trivial engineering effort. The evaluations were conducted on synthetic benchmarks, so real-world performance on noisy or incomplete knowledge graphs remains an open question. The paper also does not address latency: graph traversal adds retrieval steps that could increase end-to-end response times, particularly at scale.
What This Means for Builders
Three implications stand out for founders and engineers building RAG-based products. First, if your AI application retrieves from structured data of any kind, standard RAG is losing information by flattening hierarchies. Product catalogs, org charts, legal documents, medical records, and compliance frameworks all encode meaning in their structure. HG-RAG's graph-traversal approach offers a concrete path to improving accuracy for users navigating that structured data.
Second, the approach is framework-agnostic. The paper describes a retrieval pipeline that can be layered on top of existing RAG infrastructure including LangChain, LlamaIndex, or custom retrieval pipelines. The entity linking step, the subgraph identification, and the three-dimensional traversal can each be implemented as modular components. The bar to experimentation is relatively low for teams that already have retrieval infrastructure in place.
Third, the paper signals a broader trend. The next generation of RAG is moving from "find the right chunk" to "find the right structure." Startups that build structured retrieval infrastructure will have a meaningful edge over those treating all data as flat text. For enterprise sales, the ability to say "our AI understands your organizational hierarchy, product taxonomy, or regulatory structure" is a concrete differentiator against generic RAG solutions that treat every document like a Wikipedia page. The flat retrieval era served its purpose, but the structured retrieval era is just getting started.