A benchmark for evaluation of algorithms for identification of cellular correlates of clinical outcomes

Abstract

The Flow Cytometry: Critical Assessment of Population Identification Methods (FlowCAP) challenges were established to compare the performance of computational methods for identifying cell populations in multidimensional flow cytometry data. Here we report the results of FlowCAP-IV where algorithms from seven different research groups predicted the time to progression to AIDS among a cohort of 384 HIV+ subjects, using antigen-stimulated peripheral blood mononuclear cell (PBMC) samples analyzed with a 14-color staining panel. Two approaches (FlowReMi.1 and flowDensity-flowType-RchyOptimyx) provided statistically significant predictive value in the blinded test set. Manual validation of submitted results indicated that unbiased analysis of single cell phenotypes could reveal unexpected cell types that correlated with outcomes of interest in high dimensional flow cytometry datasets.

Computational flow cytometry: helping to make sense of high-dimensional immunology data

“…Despite its recent emergence as a sub-discipline of computational biology, researchers in flow cytometry bioinformatics have quickly realized the need for objective benchmarks and challenges to move the field forward, resulting in the FlowCAP (Flow Cytometry: Critical Assessment of Population identification methods) initiative 28,29 . This initiative serves as a forum to bring together researchers working in flow cytometry bioinformatics and to stimulate novel research directions.…”

“…Together, these results show that there is a great need for more benchmark datasets on a wider variety of problems, as many of the current FlowCAP II classification problems are relatively easy to tackle. The FlowCAP IV challenge aimed to predict the time to progression to AIDS among a cohort of HIVpositive subjects, using antigen-stimulated peripheral blood mononuclear cell (PBMC) samples 29 . The goal of this challenge was to identify novel cell popu lations that correlate well with progression and thus could be used as biomarkers.…”

Clinical Implication of Multi-Parameter Flow Cytometry in Myelodysplastic Syndromes

“…The latest challenge of "Flow-CAP," an initiative to improve computational flow cytometry data analysis, showed several algorithms that accurately predicted the time to progression from HIV to AIDS using flow cytometric data [59]. In short, there is a broad range of possibilities for automated analysis, although implementation in MDS is currently limited.…”

Integrated functional and mass spectrometry-based flow cytometric phenotyping to describe the immune microenvironment in acute myeloid leukemia

“…4). Where possible, we have implemented best practices that have been suggested by FlowCAP (Aghaeepour et al, 2016; Aghaeepour et al, 2013). Each step is visualized and each visualization can be faceted by clinical annotation to allow for exploration of questions based on the clinical annotation.…”