Tushar Chauhan

Distilling Open Reasoning Corpora into a 3B-Active MoE on Commodity GPUs: What Transfers, What Doesn't, and a Decontamination Check

We present an honest, reproducible study of budget-constrained knowledge distillation for reasoning, carried out on commodity hardware (a single node of 4x NVIDIA L40S, no NVLink) for $2,723 in GPU compute ($3,091 all-in with tax) against a $3,000 budget. We LoRA-fine-tune (QLoRA...

Distilling Open Reasoning Corpora into a 3B-Active MoE on Commodity GPUs: What Transfers, What Doesn't, and a Decontamination Check

We present an honest, reproducible study of budget-constrained knowledge distillation for reasoning, carried out on commodity hardware (a single node of 4x NVIDIA L40S, no NVLink) for $2,723 in GPU compute ($3,091 all-in with tax) against a $3,000 budget. We LoRA-fine-tune (QLoRA...