A comparison framework and guideline of clustering methods for mass cytometry data

Abstract

BackgroundWith the expanding applications of mass cytometry in medical research, a wide variety of clustering methods, both semi-supervised and unsupervised, have been developed for data analysis. Selecting the optimal clustering method can accelerate the identification of meaningful cell populations.ResultTo address this issue, we compared three classes of performance measures, “precision” as external evaluation, “coherence” as internal evaluation, and stability, of nine methods based on six independent benchmark datasets. Seven unsupervised methods (Accense, Xshift, PhenoGraph, FlowSOM, flowMeans, DEPECHE, and kmeans) and two semi-supervised methods (Automated Cell-type Discovery and Classification and linear discriminant analysis (LDA)) are tested on six mass cytometry datasets. We compute and compare all defined performance measures against random subsampling, varying sample sizes, and the number of clusters for each method. LDA reproduces the manual labels most precisely but does not rank top in internal evaluation. PhenoGraph and FlowSOM perform better than other unsupervised tools in precision, coherence, and stability. PhenoGraph and Xshift are more robust when detecting refined sub-clusters, whereas DEPECHE and FlowSOM tend to group similar clusters into meta-clusters. The performances of PhenoGraph, Xshift, and flowMeans are impacted by increased sample size, but FlowSOM is relatively stable as sample size increases.ConclusionAll the evaluations including precision, coherence, stability, and clustering resolution should be taken into synthetic consideration when choosing an appropriate tool for cytometry data analysis. Thus, we provide decision guidelines based on these characteristics for the general reader to more easily choose the most suitable clustering tools.

Hypercluster: a flexible tool for parallelized unsupervised clustering optimization

“…Hypercluster allows comprehensive evaluation of multiple hyperparameters and clustering algorithms simultaneously, reducing the allure of biased or arbitrary parameter selection. It also aids computational biologists who are testing and benchmarking new clustering algorithms, evaluation metrics and pre-or post-processing steps [10]. Future iterations of hypercluster could include further cutting-edge clustering techniques, including those designed for larger data sets [31,32] or account for multiple types of data [48].…”

“…Clustering is a particularly central challenge in the analysis of single-cell measurement data (e.g. single cell RNA-seq) due to its high dimensionality [ 8 – 10 ]. Clustering is also increasingly being used for disease subtype classification and risk stratification [ 11 – 19 ].…”

CyAnno: A semi-automated approach for cell type annotation of mass cytometry datasets

“…For instance, random sub-sampling of cells from each sample is routinely carried out to reduce computational complexity and clustering time 15 . Therefore, to overcome some of the limitations associated with unsupervised clustering, semi-automated approaches use "prior" knowledge or "ground truth" about the marker expression in each of the given cell types to annotated every cell of the unlabeled dataset 16 . Currently, only a few semi-automated approaches for cell label predictions are available, viz.…”

“…Both ACDC and SCINA, uses the list of pre-defined markers for a given cell type to annotate the unsupervised cell cluster(s) that expresses these signature markers. However, these methods depend upon the unsupervised clustering of cells and assume that the expression of target marker as binary (expressed or not expressed), which restricts their ability to classify highly similar cell sub-types, especially non-canonical cell types, that cannot be separated linearly 10,16 . Instead of defined marker lists, DeepCyTOF and LDA use manually gated cell types in form of a marker expression matrix, as training and test set, to build a Machine Learning (ML) model for the cell type prediction.…”

“…These ML methods can not only automatically learn marker-level features for each cell type, but they also demonstrated high accuracy in classifying noncanonical cell types with small population size. In fact, semi-automated methods like LDA were found to show much higher precision than ACDC and unsupervised methods in correctly predicting cell labels in the pool of gated populations 16 . However, these methods also suffer from elementary limitations, e.g.…”