Meta-analysis of (single-cell method) benchmarks reveals the need for extensibility and interoperability

Abstract

Computational methods represent the lifeblood of modern molecular biology. Benchmarking is important for all methods, but with a focus here on computational methods, benchmarking is critical to dissect important steps of analysis pipelines, formally assess performance across common situations as well as edge cases, and ultimately guide users on what tools to use. Benchmarking can also be important for community building and advancing methods in a principled way. We conducted a meta-analysis of recent single-cell benchmarks to summarize the scope, extensibility, neutrality, as well as technical features and whether best practices in open data and reproducible research were followed. The results highlight that while benchmarks often make code available and are in principle reproducible, they remain difficult to extend, for example, as new methods and new ways to assess methods emerge. In addition, embracing containerization and workflow systems would enhance reusability of intermediate benchmarking results, thus also driving wider adoption.

The current landscape and emerging challenges of benchmarking single-cell methods

“…To account for this challenge, one could perform meta-analysis or combine the results from multiple studies. However, in the current single-cell benchmarking field, the input and output of the methods are rarely made available and researchers who want to extend from existing benchmarks would need to reconstruct the benchmark from scratch (Sonrel et al 2023). This situation calls for a collective effort from all researchers for more transparency in result sharing and for the development of novel approaches to create a consensus benchmarking framework that can be extended from different methods, datasets and criteria.…”

“…In light of the rapid advancement of methodologies and the acknowledged significance of benchmarking within the single-cell research field, this field stands out as a good exemplar to explore the present state of benchmarking practices and help understand gaps that necessitate community attention. Recently, Sonrel and colleagues provided the first paper (Sonrel et al 2023) that quantitatively reviewed a collection of 62 single-cell benchmarking papers. They emphasised the technical aspects of the single-cell benchmark works and highlighted the need for code reproducibility, interoperability and extensibility.…”

Assessing the impact of transcriptomics data analysis pipelines on downstream functional enrichment results

“…To ensure that the analyses can be reproduced, extended, and scaled effectively, we developed FLOP using Nextflow [34,35]. We grouped the methods into three processing modules: (I) filtering, (II) normalisation and DE, and (III) functional analysis.…”

“…Given the rapidly evolving landscape of single-cell data analysis, it may take some time before it stabilises enough to support the use of a method like FLOP. For instance, benchmarks evaluating alternative normalisation methods for single-cell data are still under active development [35,51].…”

“…The workflow is divided into several processes. The first process downloads the three gene set collections: PROGENy [35], DoRothEA [36] and MSigDB hallmarks [37]. The second step carries out the 6 alternative DE analysis pipelines: tmm-limma, vsn-limma, voom-limma, log quantile-limma, edgeR and DESeq2.…”