iSAGE: A Human-in-the-Loop Framework for Remote Sensing Semantic Segmentation via Sparse Point Supervision

Abstract

Training semantic segmentation models is especially costly in remote sensing, since most problems require a new dataset as targets vary with resolution, sensors, and region. Approaches to reduce this cost are increasingly common, but most add machinery that expands few labeled pixels into a denser training signal using the model’s own predictions, where a confident prediction looks the same whether it is correct or incorrect. The hypothesis here is that these confident errors are the most valuable pixels to label, and that a human examining the image can identify them directly. We propose iSAGE (Iterative Sparse Annotation Guided by Expert), an open-source framework in which the annotator clicks confident errors in the prediction overlay, the clicks train the model under an error-weighted loss, and the updated model surfaces the next errors, with no automatic label expansion at any step. With at most one labeled pixel per class per frame per iteration, iSAGE recovers 96.9% of dense performance on BsB Aerial (74.79% mIoU from 0.040% of the pixels) and matches the dense baseline on ISPRS Vaihingen (76.65% vs. 76.93% from 0.011% of the pixels), surpassing published weakly supervised methods by nearly 4 mIoU points. Four automatic selection strategies run through the same loop plateau 7.3 to 14.4 points below iSAGE, and raising their budgets far higher does not close the gap; in a 35-method comparison, iSAGE is the only iterative human-in-the-loop framework without automatic label generation.