Process-Model-Supported Offline Reinforcement Learning for Water–Fertilizer Management: Two-Model Closed-Loop Production–Nitrogen-Loss Trade-Offs

Abstract

Nitrogen leaching threatens groundwater quality, making production retention and efficient water–fertilizer use joint management priorities. Field trials evaluate a limited set of treatments, while adaptive policies generate long sequences of decisions. We propose counterfactual rollout-constrained offline reinforcement learning (CR-ORL), a framework that connects process-model simulation, executed sequential data, history-aware learning, and closed-loop evaluation. Bellman training uses only transitions with realized continuations, and policies select actions without access to candidate outcomes. Under this common protocol, a supervised Ranker-Selected baseline was compared with implicit Q-learning (IQL), conservative Q-learning (CQL), and batch-constrained Q-learning (BCQ). Policies trained with five seeds were evaluated on two held-out lineages for each of the VEG and PS123 configurations. In these simulations, relative to the original-schedule control, Ranker-Selected reduced total nitrogen leaching by 98.51% in VEG and 62.12% in PS123, with production reductions of 12.79% and 6.18%, respectively. IQL, CQL, and BCQ retained more production: their production reductions were 1.82–2.30% in VEG and 3.05–3.57% in PS123, alongside nitrogen-leaching reductions of 62.08–66.80% and 39.46–44.11%. Ranker-Selected also reduced irrigation and fertilizer most. Additional VALIDATION comparisons showed that simple reduction rules remained competitive in several production–burden projections. Post hoc retraining under alternative reward priorities changed behavior without consistent shifts toward the intended physical outcomes. CR-ORL exposes these distinct production–resource–nitrogen-loss trade-offs within two process-model configurations.