Computational methods for benchmarking and development of best practices for virus discovery and characterization in the HVP consortium - The Human Virome Program (HVP) seeks to advance our understanding of the human virome’s role in health and disease, but the field faces persistent challenges in virus discovery and characterization due to inconsistencies in analytic methods and uneven access to tools and benchmarking datasets. Currently, a range of bioinformatic tools exist for viral genome assembly, taxonomic classification, and novel virus identification, although many require specialized computing environments or expertise. Moreover, there is a lack of standardized benchmarks for assessing tool performance, leading to fragmented evaluations that make it difficult for researchers to identify the most effective methods for their specific needs. This absence of uniform evaluation criteria undermines reproducibility and reduces the efficiency of methodological innovation. To address these challenges, we propose the development of a platform to provide access to: 1) high-priority analytic tools, 2) standardized benchmarking datasets, and 3) an environment for continuous tool evaluation. Leveraging our expertise in genomic data analysis and interoperable computing, we will first establish a common HVP tool registry, prioritizing bioinformatic methods critical for virome research, including viral genome assembly, taxonomic assignment, and novel RNA virus identification. These tools will be ported to the Dockstore tool registry to maximize availability to all HVP researchers. This collection will be continuously maintained based on feedback from the HVP Bioinformatics Working Group and tool authors. To support standardized evaluation, we will construct comprehensive benchmarking datasets that reflect the complexity and diversity of human virome samples. These datasets will include both real and synthetic data, spanning a range of virome compositions, host backgrounds, and sequencing technologies. Synthetic datasets will be designed to test specific tool capabilities under controlled conditions, incorporating known viral genomes, engineered mutations, and varying levels of host and microbial contamination. Real datasets will be curated from validated sources, providing examples for benchmarking common analytic tasks. These resources will allow for standardized, reproducible comparisons of tool performance across multiple metrics, including accuracy, sensitivity, precision, recall, computational efficiency, and resource utilization. Finally, we will develop a benchmarking infrastructure using shared workspaces on the Terra cloud compute platform that integrate benchmarking datasets with analytic workflows, enabling users to test and compare tools directly. Researchers will have the flexibility to customize benchmarking parameters and compare performance across different use cases. To foster community engagement and continuous improvement, we will coordinate two benchmarking challenges officiated by the HVP Bioinformatics Working Group, the first involving a single use case of known virus identification, and the second spanning three additional priority use cases: 1) novel virus identification, 2) phage profiling, and 3) host prediction.
