A metric and workflow for quality control in the analysis of heterogeneity in phenotypic profiles and screens

Abstract

Heterogeneity is well recognized as a common property of cellular systems that impacts biomedical research and the development of therapeutics and diagnostics. Several studies have shown that analysis of heterogeneity: gives insight into mechanisms of action of perturbagens; can be used to predict optimal combination therapies; and to quantify heterogeneity in tumors where heterogeneity is believed to be associated with adaptation and resistance. Cytometry methods including high content screening (HCS), high throughput microscopy, flow cytometry, mass spec imaging and digital pathology capture cell level data for populations of cells. However it is often assumed that the population response is normally distributed and therefore that the average adequately describes the results. A deeper understanding of the results of the measurements and more effective comparison of perturbagen effects requires analysis that takes into account the distribution of the measurements, i.e. the heterogeneity. However, the reproducibility of heterogeneous data collected on different days, and in different plates/slides has not previously been evaluated. Here we show that conventional assay quality metrics alone are not adequate for quality control of the heterogeneity in the data. To address this need, we demonstrate the use of the Kolmogorov-Smirnov statistic as a metric for monitoring the reproducibility of heterogeneity in an SAR screen, describe a workflow for quality control in heterogeneity analysis. One major challenge in high throughput biology is the evaluation and interpretation of heterogeneity in thousands of samples, such as compounds in a cell-based screen. In this study we also demonstrate that three heterogeneity indices previously reported, capture the shapes of the distributions and provide a means to filter and browse big data sets of cellular distributions in order to compare and identify distributions of interest. These metrics and methods are presented as a workflow for analysis of heterogeneity in large scale biology projects.

Biologically Relevant Heterogeneity: Metrics and Practical Insights

“…Quadratic entropy has been applied to describe the diversity in cell populations. 4,37, 76, 134 Shape measures often use a normal distribution as a reference and make a qualitative or quantitative comparison with the data.…”

“…Finally, the tail of the distribution can be characterized by the outliers. The percent outliers in the population 4, 37 can indicate whether the population has a normal or more heavy-tailed distribution. 131 …”

“…37 It is important to note that metrics such as the Z’-factor, or the SSMD give no information about the consistency of the distributions in the wells. 37 Figure 4 illustrates a workflow for heterogeneity analysis that addresses the need for metrics and quality control. The suggested procedure follows the same principles used for quality control in screening and therefore integrates well with a standard screening protocol.…”

“…The procedure adopts a new metric, the QC-KS (Figure 4, steps 2 & 3) that uses the KS statistic to compare the distributions in the control wells on each plate to a set of reference distributions established during validation. 37 The QC-KS metric ensures that the shape of the control distributions is consistent throughout the project.…”

“…4, 37–39 The importance of minimizing the ‘system variability’ is critical to achieving consistent, quantitative measurements, as has been discussed in detail for high content imaging and flow cytometry. 40–42 Flow cytometry has a long history and mature process for system calibration, characterization and standardization, including published protocols 43 and an array of reference standards.…”