HiRAG: A Hierarchical Retrieval-Augmented Generation Framework via Intent Routing and Small-Model Refinement

Abstract

While retrieval-augmented generation (RAG) mitigates large language model (LLM) hallucinations, traditional flat vector retrieval in massive, heterogeneous databases suffers from high latency and noise interference. To address this, we propose HiRAG, a hierarchical RAG framework. Offline, HiRAG builds a “classification-summary” dual-layer index to structurally organize data. Online, a lightweight bidirectional encoder representations from transformers (BERT)-based router identifies query intent to dynamically prune the search space. A cascaded pipeline then uses hybrid retrieval and Small-LLM reranking for coarse-to-fine de-noising. Experiments on the Microsoft Machine Reading Comprehension (MS MARCO) and Multi-Domain Enterprise Knowledge Base (MDE-KB) datasets demonstrate that HiRAG outperforms traditional baselines. Specifically on MDE-KB, HiRAG improves Recall@5 by 14.5 percentage points over Naive RAG and achieves a retrieval augmented generation assessment (RAGAS) Answer Relevance score of 0.92, while reducing average query latency by 28%. HiRAG provides a viable, efficient solution for large-scale knowledge bases.