Benchmarking short-read germline structural variant calling highlights advantages of using ensembles of tools and small impact of graph genome alignment

Abstract

Structural variants (SVs) in the human genome play an important role in health and disease. Identification of SVs is commonly performed using short-read sequencing. However, because sequenced fragments are typically shorter than the variants themselves, accurate SV calling remains a challenging problem. Previous benchmarking studies consistently find substantial variability in performance among SV calling tools, with disagreements largely driven by differences in underlying algorithms. We evaluate 14 high-performing SV calling tools using four samples from a newly published, high-confidence pedigree-based truth set, together with an in-house dataset. Its recent release minimizes the likelihood that it was used for training by tools, thereby reducing the risk of over-fitting. We implemented frequency filtering that reduced the downstream variants interpretation load by nearly half. We also assess alignments from a graph genome assembly, but found only small effect on performance, compared to a linear reference. DRAGEN achieves the highest overall performance, with a mean recall of 0.36 and mean F1 score of 0.51. Among the open-source tools, Manta and Dysgu achieve the highest recall, both with a mean recall of 0.26, while Manta achieves the highest mean F1 score of 0.41. We also evaluate several multi-caller ensemble strategies, which in some settings achieved higher recall than any individual tool. Across both ensemble strategies, Dysgu, Octopus, Manta, and Tardis are part of the top-performing sets. Our findings highlight the importance of designing SV calling strategies using one or more tools according to the intended application, while balancing accuracy, recall, computational resources, reference choice, and clinical interpretation burden.