Budgeted Reward Allocation: A Position on Unifying Sparse, Verifiable, and Hierarchical Reward Signals in LLM Post-Training

Abstract

Reward hacking, verbosity, and hallucination are usually treated as separate failure modes in the post-training of large language models, addressed by separate mechanisms: reward shaping for verbosity, verifiable reward models (RLVR) for hallucination, hierarchical decomposition for long-horizon tasks, and load-balancing losses for mixture-of-experts (MoE) routing. This position paper argues that these four mechanisms can be described as instances of a single accounting abstraction: a finite, per-task reward budget that is released only against verified sub-goal completion, decays to a penalty once exhausted, and can be propagated across a task hierarchy and an expert pool as a routing signal. We do not claim that any individual component is new — verifiable rewards, reward shaping, hierarchical reinforcement learning, and MoE routing are each established, actively studied techniques, and we cite the relevant literature for each. The contribution we argue for is narrower and more honest: a shared bookkeeping formalism (budget allocation, exhaustion, and propagation) that makes explicit how these four techniques already interact in practice, and that suggests a small number of concrete, testable predictions about where combining them should help or hurt.