Decomposing Apparent Skill in Machine-Learning Event Classification: A Multi-Catalog Study of Label Structure, Feature Coupling, and Operational Generalization

Abstract

Machine-learning evaluation assumes that a held-out target represents the decision problem of interest. That assumption is difficult to satisfy when labels are constructed retrospectively from the same observational record that supplies the predictors, because a classifier can then reproduce regularities embedded in the target-generation procedure rather than resolve the harder decision. We treat earthquake event-role classification as an experimental case study of this problem and develop a diagnostic decomposition of apparent skill. Three catalogs differing in size, completeness, and feature availability are evaluated with chronologically partitioned gradient-boosted classifiers under a common window-defined target, and conventional held-out performance is decomposed using structural magnitude baselines, feature-family ablation, evaluation conditioned on the information state, synthetic epidemic-type-aftershock-sequence procedural nulls, alternative target definitions, probability calibration, conformal prediction, and cross-catalog transfer. Pooled discrimination was high, at ROC AUC 0.889 to 0.972, but fell by 0.264 to 0.387 when evaluation was restricted to events whose role is unresolved by preceding local magnitude ordering. A single magnitude-difference statistic came within 0.006 of the fitted models in two catalogs and exceeded them in the third, and removing the magnitude feature family reduced discrimination by 0.10 to 0.16. Synthetic catalogs generated without parent–offspring magnitude dependence reproduced both the high pooled score and its contraction under conditioning. Calibration, conformal coverage, and transfer behaviour changed independently of discrimination. The contribution is a reproducible diagnostic framework for interpreting machine-learning performance on retrospectively constructed scientific targets.