Joint Optimization of Retrieval-Augmented KV Caching and Adaptive Speculative Decoding for Autoregressive Generation

Abstract

Abstract Retrieval-assisted key–value (KV) reuse and speculative decoding remove different costs, but their naive composition can leave draft and verification overhead after prefill is shortened. RAKV-ASD combines exact causal prefix validation with a measured controller that selects reuse and k∈{0,1,2,4,6,8}. Across 3,600 timed executions on two GPT-2-family pairs, every greedy output matched target-only decoding. Pooled results show 64.0% lower latency and 170.8% higher throughput than reuse plus fixed k=4, while remaining within 0.45% of the KV-only baseline. The controller selected k=0, achieved 100% k-decision accuracy and 0.32% mean regret despite 32.1% point-prediction MAPE because costs rank actions rather than provide calibrated latency forecasts. Sensitivity and a third-pair study place the nonzero-k crossover at predicted acceptance ≥0.894. The result is a measured boundary and a mechanism that avoids counterproductive speculation, not a universal speedup.