BenchHub enables an inclusive and transparent ecosystem for community-focused benchmarking in computational biology

Abstract

Abstract Background The rapid growth of computational methods for the computational biology field highlights the critical role of benchmarking in guiding method selection. However, there is no standardised data structure that effectively links and stores datasets, performance metrics and available ground truth. Without such a unified and shareable structure, it is difficult for the community to contribute, update and extend existing benchmarking studies to ensure long-term relevancy. Results To address this challenge, we present BenchHub, a community-oriented ecosystem with a modular R6-based structure that enables “living benchmarking”. BenchHub comprises three key components: a Trio database that links datasets, performance metrics, and supporting evidence (e.g. ground truth); a BenchmarkStudy structure that captures the different benchmark study designs; and a series of tools together with vignettes and interactive platform that allow users to gain insights from the benchmarking results. Conclusions Together, these components streamline the benchmarking process for benchmark study developers, methods contributors, and benchmark consumers, promoting reproducibility, comparability, and long-term sustainability in computational biology.