HierCache-RAG: Fast Retrieval-Level Caching for GraphRAG

Abstract

Graph Retrieval-Augmented Generation (GraphRAG) improves multi-hop reasoning in Large Language Models (LLMs) by grounding generation in knowledge-graph structure, but most systems re-traverse the graph on every query, wasting computation whenever queries repeat or overlap semantically. We propose HierCache-RAG, a hierarchical retrieval-level caching framework that reuses graph-derived relational context through three cascading layers – an Exact Query Cache, a Semantic Query Cache, and a One-Hop Graph Cache – before falling back to graph traversal. Unlike prior work that caches at the generation or key-value stage, HierCache-RAG caches directly at the retrieval stage. By progressively reusing previously retrieved relational contexts, the framework avoids repeated graph traversal for both exact and semantically similar queries. On DBLP, it reaches 2.03 ms effective retrieval latency (2.32×/10.70× faster than NetworkX/Neo4j) with competitive answer quality; on the more semantically diverse arXiv dataset, it reaches 1.29 ms (up to 61.43× faster) while also achieving the highest answer quality of any system tested. These results indicate that retrieval-level caching can make interactive GraphRAG systems practical for larger knowledge graphs without sacrificing response quality. The performance gains remain stable as the graphs scale from 30K to 70K papers, and the approach is complementary to existing generation-level and KV caching techniques.