Exact Layer Streaming: LoRA Fine-Tuning of an 8B Model on a 4 GB Laptop GPU

Abstract

Parameter-efficient fine-tuning is bounded by a hard constraint: the frozen base model must fit in GPU memory. Layer streaming — holding a small pool of decoder layers in VRAM and fetching the rest from host RAM on demand — removes that constraint in principle, but the published systems that stream weights during training target datacenter hardware: an H200 with 1.5 TB of host memory, an RTX 4090 with 256 GB. Systems that reach a 4 GB laptop either do not train or do not stream the base; the published result for that class is 1.3B, by projection, not streaming. We report layer-streamed LoRA training on a 4 GB laptop GPU at two frontiers: Llama-3.1-8B in NF4 at 119.6 tok/s with a 3.32 GB peak, and Qwen2.5-3B with an un-quantized bf16 base at 143.1 tok/s in 2.15 GB — a configuration that raises CUDA out of memory when trained resident on the same card. Both NF4 throughput figures predate the repair below and were not re-run on that card; correctness was never affected. Overhead is 1.43× at 0.5B, the only size with a valid resident baseline. Because streaming failures are silent — a severed autograd path still yields a falling loss — the central contribution is a correctness protocol gated on bit-exactness against a resident reference of the same numerics. Version 2 ran it at real sizes and this version changes nothing there: the forward is torch.equal from 0.5B to 72B, the backward exact at 8B and 14B. Above that it caught a second silent defect, this one upstream: at 32B and 72B in NF4 the forward stayed bit-exact and the loss matched resident to every digit while the gradients were wrong on 62 of 64 layers at 32B and 78 of 80 at 72B. The cause is aliasing, not a race; it is filed upstream, repaired at −4.8% throughput, and re-gated against a control that reproduced it in the same process. No number here changes: the defect lives above the size this paper claims, and we found it only by looking there. The 8B configuration reproduces on an H100 at a median 113.00 tok/s in the same peak: whatever bounds the step is common to both machines and is not the compute the datacenter card adds. Version 2 read that as evidence the method is bound by host-to-device transfer; a later probe on the laptop measured it and refuted it — the step runs at 71.3% of that card's same-session GEMM ceiling, deleting every host-to-device byte buys 1.4%, and the largest streaming-specific cost is the per-layer NF4 dequantisation at 9.8%. No measured number changes. Downstream quality is indistinguishable from resident at roughly 1pp resolution. Four findings are streaming-independent: the cross-entropy logits term costs 14 bytes per element, not the 6 first principles suggest; Windows/WDDM spills into host memory instead of raising OOM, so a completed run is not evidence of fit; gradient accumulation is per-token I/O-neutral; and a fraction-of-peak figure on a consumer GPU is meaningless unless the throughput and the ceiling were measured in the same session with the SM clock recorded.