Retrieval-Augmented Generation with Hierarchical Knowledge

HiRAG is a novel Retrieval-Augmented Generation framework that incorporates hierarchically structured knowledge into both the indexing and retrieval phases to improve LLM performance on domain-specific tasks. By mimicking the hierarchical nature of human cognition, HiRAG enhances semantic understanding and structural awareness beyond flat or graph-based RAG methods.

Problem Statement

Existing graph-based RAG methods fail to leverage the naturally hierarchical organization of knowledge as it exists in human cognition, leading to suboptimal semantic understanding and structural capture. This limits RAG systems' ability to reason across different levels of abstraction (e.g., concepts, topics, documents). Addressing this gap is critical for improving LLM reliability and depth of reasoning in specialized domains.

Key Novelty

Introduction of hierarchical knowledge structures into both the indexing and retrieval pipelines of a RAG system, going beyond flat or single-level graph representations
A new indexing strategy that organizes knowledge at multiple levels of granularity to better reflect cognitive hierarchies (e.g., entity, concept, document levels)
A retrieval mechanism that navigates hierarchical knowledge graphs to surface contextually relevant information at the appropriate level of abstraction

Evaluation Highlights

HiRAG achieves significant performance improvements over state-of-the-art graph-based RAG baselines across domain-specific benchmarks (specific deltas not reported in abstract)
Extensive experiments validate improvements in both semantic understanding and structural capturing capabilities during indexing and retrieval

Breakthrough Assessment

6/10 HiRAG presents a well-motivated and practically useful extension of graph-based RAG by introducing hierarchical knowledge structures, which is a meaningful contribution to the RAG ecosystem. However, the core idea of leveraging hierarchy is an incremental architectural improvement rather than a paradigm shift in how LLMs are augmented with external knowledge.

Methodology

Hierarchical Indexing: Documents and knowledge are organized into multi-level hierarchical structures (e.g., entities → concepts → topics) during the indexing phase, capturing relationships at varying levels of abstraction
Hierarchical Graph Construction: A knowledge graph is built that encodes both intra-level and cross-level relationships, enabling traversal across abstraction layers during retrieval
Hierarchical Retrieval: At query time, the system navigates the hierarchical graph to retrieve knowledge at the most semantically appropriate granularity level before passing context to the LLM for generation

System Components

Hierarchical Indexer

Organizes raw documents into a multi-level knowledge structure during preprocessing, capturing semantic relationships at different granularities

Hierarchical Knowledge Graph

A graph representation that encodes entities, concepts, and topics as nodes across multiple layers with edges representing both same-level and cross-level relationships

Hierarchical Retriever

A retrieval module that traverses the hierarchical graph in response to a query, selecting relevant knowledge at the appropriate abstraction level

LLM Generator

A large language model that receives hierarchically retrieved context to produce accurate, grounded responses for domain-specific tasks

Results

Metric/Benchmark	Best Baseline (Graph RAG)	HiRAG	Delta
Domain-specific QA accuracy	State-of-the-art baseline	Significantly higher	Positive improvement
Semantic understanding quality	Graph-based RAG	Enhanced via hierarchy	Qualitative gain
Structural capturing capability	Flat/graph RAG	Improved multi-level capture	Qualitative gain

Key Takeaways

Incorporating hierarchical knowledge organization into RAG pipelines—mirroring human cognitive structures—can meaningfully improve LLM performance on domain-specific tasks without retraining the base model
When building RAG systems for specialized domains, consider multi-granularity indexing (entity → concept → topic) rather than flat chunk retrieval or single-level graphs to better capture contextual depth
HiRAG's approach suggests that the retrieval strategy should be query-adaptive in terms of abstraction level, making the retriever a key component to invest in for production RAG systems

Abstract

Graph-based Retrieval-Augmented Generation (RAG) methods have significantly enhanced the performance of large language models (LLMs) in domain-specific tasks. However, existing RAG methods do not adequately utilize the naturally inherent hierarchical knowledge in human cognition, which limits the capabilities of RAG systems. In this paper, we introduce a new RAG approach, called HiRAG, which utilizes hierarchical knowledge to enhance the semantic understanding and structure capturing capabilities of RAG systems in the indexing and retrieval processes. Our extensive experiments demonstrate that HiRAG achieves significant performance improvements over the state-of-the-art baseline methods.