A finite-sample inferential framework for the weighted kappa coefficient (FWKC) in binary diagnostic agreement studies

Abstract

Abstract Background The weighted kappa coefficient κ(c), where c is the relative loss of a false-negative result, summarizes beyond-chance agreement between a binary diagnostic test and a reference standard. Established intervals for κ(c) may be unreliable in small samples and under prevalence imbalance. We developed and evaluated a resampling framework (FWKC). Methods κ(c) was derived from a conventional weight matrix. FWKC draws sensitivity and specificity from Jeffreys beta posteriors, generates pseudo-rater tables, pools their kappa values with a crossed Bayesian model and a leave-one-rater-out random forest, fuses both by inverse-standard-error weighting, and quantifies uncertainty with a full-pipeline percentile bootstrap. The primary simulation comprised 243 scenarios with n = 30–100 and prevalence and c from 0.1 to 0.9, with 120 independent datasets per scenario. A separate supportive expanded robustness tier used the full prespecified 162-cell 3 × 3 stress grid, crossing the same six non-anchor data-generating mechanisms with n ∈ {30,50,100}, p ∈ {0.1,0.5,0.9}, and c ∈ {0.1,0.5,0.9}. Each scenario used M = 40 independent datasets, with B = 49 inner bootstrap resamples within each dataset. Thirty-four sensitivity variants and two real-data applications provided further checks. Results The implementation closely reproduced the published intervals for the Weiner et al. data. In the 243-scenario primary grid, the SE-expanded FWKC interval alone avoided coverage below 0.85, reaching at least 0.90 coverage in 79 of 81 scenarios at n = 30, at the cost of a substantially greater width. The transformed (Fisher-z) delta-method interval remained the best-calibrated inexpensive option in the primary grid. In the 162-scenario expanded robustness tier, the SE-expanded interval had mean successful-fit coverage of 0.955, a mean absolute calibration error of 0.032, and coverage below 0.85 in only 1 of 162 scenarios, but its mean width was 0.903 versus 0.505 for the transformed interval. For point estimation, the random-forest component had the lowest mean RMSE across all six DGM groups (0.143 overall, versus 0.149 for the direct estimator, 0.159 for the Bayesian component, and 0.154 for the fused estimator). Conclusions The transformed delta-method interval remains the inexpensive default for routine inference, whereas the SE-expanded FWKC interval provides a deliberately wider option when protection against severe small-sample under-coverage is the priority. The expanded robustness analysis strengthens support for the SE-expansion layer and for random-forest aggregation, but not for the hybrid fusion: the random-forest component was the most accurate component on average across all six DGM groups, while the fused estimator did not improve on it overall. The two inferential strategies are therefore complementary rather than universally ordered. Clinical trial number Not applicable.