Measuring error rates in genomic perturbation screens: gold standards for human functional genomics

Abstract

Technological advancement has opened the door to systematic genetics in mammalian cells. Genome-scale loss-of-function screens can assay fitness defects induced by partial gene knockdown, using RNA interference, or complete gene knockout, using new CRISPR techniques. These screens can reveal the basic blueprint required for cellular proliferation. Moreover, comparing healthy to cancerous tissue can uncover genes that are essential only in the tumor; these genes are targets for the development of specific anticancer therapies. Unfortunately, progress in this field has been hampered by off-target effects of perturbation reagents and poorly quantified error rates in large-scale screens. To improve the quality of information derived from these screens, and to provide a framework for understanding the capabilities and limitations of CRISPR technology, we derive gold-standard reference sets of essential and nonessential genes, and provide a Bayesian classifier of gene essentiality that outperforms current methods on both RNAi and CRISPR screens. Our results indicate that CRISPR technology is more sensitive than RNAi and that both techniques have nontrivial false discovery rates that can be mitigated by rigorous analytical methods.

An extended set of yeast-based functional assays accurately identifies human disease mutations

“…Unfortunately, it can be challenging to identify a phenotype in human cells that is strong enough to quantitatively evaluate functional variation. Based on a set of high-performing shRNA screens across different human cancer cell lines (Marcotte et al 2012), Hart et al compiled a list of 291 genes that showed some growth phenotype in at least half of 48 cell lines (Hart et al 2014). Only one (4%) of the 26 complementing human disease genes identified in this study yielded a fitness effect in Hart et al Even accepting cases where the phenotype was observed in only a single cell line within either of two large-scale shRNA screens (Cheung et al 2011;Marcotte et al 2012) (and assuming that all of these phenotypes are strong enough to form the basis of a robust functional assay), only 13 (50%) of the 26 complementing human genes in this study showed any shRNA phenotype.…”

Natural Variation in Gene Expression Modulates the Severity of Mutant Phenotypes

“…Gene-expression and siRNA knockdown data for human cell lines were obtained from the COLT database (http://colt.ccbr.utoronto.ca/cancer/) and analyzed based on Z scores (Marcotte et al, 2012) and on Bayes Factors (Hart et al, 2014). We identified genes that are required for normal cell growth by fitting a logistic regression model from the COLT RNAi Bayes Factors in different cell lines to a set of genes whose orthologs are known to be required for viability in C. elegans and S. cerevisiae.…”

ScreenBEAM: a novel meta-analysis algorithm for functional genomics screens via Bayesian hierarchical modeling

“…Housekeeping or orthologous genes that were not present in the shRNA library were filtered out. In additional to the above knowledge-based house-keeping genes, we also used a recent essential gene list identified from genome-wide RNAi screens of 48 cancer cell lines (Hart et al, 2014). We selected genes (N ¼ 272) that were identified as essential in over 24 (50%) out of 48 cell lines as an independent gold standard reference of essential genes (Supplementary Table S1).…”

“…For NGS-based count, we employed log-normal distribution and performed log2-transformation of count values before fitting the above model To evaluate the performance of this ScreenBEAM algorithm, compared with classical RIGER (RIGER_KS, RIGER_SB, RIGER_WS) and RSA methods we tested the overlap of their predictions with the gold standard housekeeping or orthologous genes set (see description in methods section), using three benchmark shRNA screens from the microarray-based dataset (Marcotte et al, 2012). For these studies we did not use the essential gene list identified from genomewide RNAi screens (Hart et al, 2014) to prevent over-fitting due to the use of the same dataset. For the RIGER and RSA methods, we defaulted to the t-statistics to score individual shRNA hairpins (as implemented in their corresponding software packages).…”

“…For example, many candidate therapeutic targets identified from shRNA screens have failed validation in independent assays (Babij et al, 2011;Begley and Ellis, 2012;Prinz et al, 2011). Among the possible reasons for lack of robustness, one must consider RNAi off-target effects, variable potency and knock-down/out efficiency of RNAi and CRISPR reagents, biological and technical noise in highthroughput screens (Echeverri et al, 2006;Hart et al, 2014;Shao et al, 2013), cell line-specific efficiency of viral pool infection and toxicity from additional viral vector expression cassettes (e.g. fluorescence reporter proteins).…”