LawSaathi-RAG: A Benchmark for Comparing Retrieval-Augmented Generation Architectures on Indian Legal Question Answering

Abstract

Access to accurate legal information remains a critical challenge for Indian citizens, particularly those navigating complex statutory and case-law frameworks. Retrieval-Augmented Generation (RAG) offers a promising approach by grounding large language model (LLM) outputs in authoritative source documents. However, the choice of retrieval architecturelexical, dense semantic, or hybrid ensemblesignificantly impacts answer quality and latency. In this work, we present LawSaathi-RAG, a reproducible open-source benchmark comparing three RAG architectures on 200 question-answer pairs derived from Indian Supreme Court judgments. We implement NaiveRAG (BM25 lexical retrieval), DenseRAG (sentence-BERT embeddings with cosine similarity), and HybridRAG (weighted ensemble of BM25 and dense scores). Our custom evaluation suite measures Exact Match, Token F1, Precision@k, Recall@k, and end-to-end latency. We find that BM25-based lexical retrieval outperforms dense semantic retrieval on legal domain text (Precision@5: 0.644 vs. 0.375), while HybridRAG achieves the lowest latency (1,898 ms). Ablation studies on chunk size (256/512/1024 tokens) reveal diminishing returns from fine-grained chunking. Error analysis on 30 failure cases identifies a critical preprocessing gap: retrieved chunks frequently contain case metadata (titles, party names) rather than substantive legal reasoning, causing the LLM to correctly abstain rather than hallucinate. All code, data, and evaluation scripts are publicly available at https://github.com/SidakSethi-Singh/LawSaathi-RAG.