How much of a Dice score is the reference? A pre-registered blind audit of ISIC 2017 lesion masks against an independent tracer, a trained network, a training-free rule and a blinded preference panel

Abstract

Project page: https://deolhonapele.com.br · Try the contour challenge yourself (open, anonymous, nothing recorded): https://deolhonapele.com.br/desafio-contorno/aberto/ Background. Skin-lesion segmentation methods are ranked by their agreement (Dice) with a single human reference mask per image. Where part of that reference is coarse, the Dice mixes accuracy with agreement with the coarseness. Whether networks trained on such references inherit their errors is rarely tested. Methods. The study used the 580 ISIC 2017 Test images that were not touched during development. Before any result was opened, the following were frozen with SHA-256 hashes: (i) a training-free, clinician-written rule that delineates pigmented lesions or refuses with a stated cause; (ii) a public network trained on ISIC 2017 Training (network A); (iii) a U-Net/ResNet-34 trained by the author on ISIC 2017 Training only (network B); (iv) the blind tracings of an independent lay tracer (the author's brother), who saw only the photographs; (v) a blind audit of every reference mask, judged correct, wrong (border visible but not followed) or non-assessable (border not visible), by the author, a nurse, who saw no algorithm output and no number. Sixteen addenda and nine falsifiable predictions were registered. Afterwards, a blinded panel of invited health professionals compared the reference with each alternative outline in 240 forced-choice screens (Addendum 11D). Results. Audit: the pre-registered primary outcome, rule × reference, was a median Dice of 0.807 (95% CI 0.760–0.830), below the 0.886 obtained during development on ISIC 2018. The audit judged 282 references correct (49%), 239 wrong (41%; 231 of them coarse) and 59 non-assessable (10%). Tracer: the independent tracer agreed with the reference at a median Dice of 0.901 where the reference was judged correct and 0.731 where it was judged wrong. On the wrong ones, the reference was larger than the tracer's outline in 98% of images. In exploratory analyses, the disagreement was dominated by extent rather than location. Network B: it agreed with the reference at 0.903 overall. On the wrong references it stayed closer to the tracer (0.864) than the reference itself was (0.731). Rule: it refused 17.4% of images (scored zero), four in five of them because the lesion touched the frame (52) or was very large (28), and stayed below network B in every audit stratum. Where the author's audit judged the reference correct, the rule reached 0.905 (0.923 on the images it accepted), against 0.947 for network B, with no training data. Panel: twelve raters chose network B over the reference in 64% of decided screens with wrong references (95% CI 50–76%, p = 0.058). The tracer was chosen in 55% (p = 0.58) and the rule in 43% (p = 0.37). A similar, non-significant tendency appeared on correct references (12 of 16 decided screens, 75%, p = 0.077). Conclusions. In a single blind audit by one non-specialist auditor, about four in ten reference masks in this benchmark were judged coarse, mostly too wide. An independent tracer's outlines were consistent with that judgement. A strong network trained on these references reproduced little of that excess. A small, exploratory panel of professionals leaned towards the network's outlines but did not reliably separate any outline from the reference. Where the reference was judged correct, a training-free rule came within about 0.04 Dice of the trained network (0.905 vs 0.947). Benchmarks should report an audit of the reference, an independent tracer, and area, position and border distance beside any Dice. Dermatologist re-audit and a larger panel are pre-registered and pending.