How Chunk Size and Embedding Model Choice Affect Retrieval-Augmented Generation Quality and Cost: A Controlled Comparison on Consumer Hardware

Abstract

This study investigates how chunk size and embedding model choice affect the retrieval quality, end-to-end answer quality, and indexing cost of Retrieval-Augmented Generation (RAG) systems. A controlled 3 × 3 experiment evaluates three chunk sizes (128, 256, and 512 words) with three open-source embedding models: BAAI/bge-small-en-v1.5, intfloat/e5-small-v2, and all-MiniLM-L6-v2. Experiments use an 800-question subset of the Stanford Question Answering Dataset (SQuAD 1.1). Retrieval performance is evaluated using Recall@5, while end-to-end question-answering performance is measured using token-level F1 and Exact Match (EM). Indexing cost is evaluated using indexing time, peak GPU memory (VRAM), and on-disk storage. Answer generation is performed using a locally served Llama 3.2 3B model on an NVIDIA GeForce GTX 1650 Ti with 4 GB VRAM. Results show that embedding model choice has a stronger effect on retrieval quality than chunk size. E5-small achieves the highest mean Recall@5 (0.938), followed by BGE-small (0.892) and MiniLM (0.880). The highest Recall@5 is obtained with 512-word chunks and E5-small (0.946). However, retrieval quality and end-to-end answer quality are only weakly correlated (r = +0.12), indicating that higher retrieval performance does not necessarily produce proportionally better generated answers. The highest F1 score is 0.415 for 512-word E5-small, while 256-word MiniLM provides the highest Exact Match rate (0.22). MiniLM also provides the lowest indexing cost. The findings demonstrate that chunk size and embedding model selection involve different quality-cost trade-offs in RAG systems. The results also highlight the importance of evaluating RAG pipelines using both retrieval-level and end-to-end generation metrics rather than relying on retrieval performance alone. Keywords: Retrieval-Augmented Generation, RAG, chunk size, embedding models, dense retrieval, information retrieval, question answering, SQuAD, sentence embeddings, large language models, LLM, retrieval evaluation.