Yehoon Choi

Title RER-Q: Reading a Quantized KV Cache with Fresh Samples at Every Decoding Step

In long-context decoding the model reads its whole key–value (KV) cache at every step, and the bytes read per step set the step time, most of all when the cache is offloaded to host memory and read over PCIe. We present RER-Q (Resample-Every-Read with Quantized storage). It store...

Title RER-Q: Reading a Quantized KV Cache with Fresh Samples at Every Decoding Step

In long-context decoding the model reads its whole key–value (KV) cache at every step, and the bytes read per step set the step time, most of all when the cache is offloaded to host memory and read over PCIe. We present RER-Q (Resample-Every-Read with Quantized storage). It store...

Title RER-Q: Reading a Quantized KV Cache with Fresh Samples at Every Decoding Step

In long-context decoding the model reads its whole key–value (KV) cache at every step, and the bytes read per step set the step time, most of all when the cache is offloaded to host memory and read over PCIe. We present RER-Q (Resample-Every-Read with Quantized storage). It store...