Evaluation of the discriminative accuracy of genomic profiling in the prediction of common complex diseases

Moonesinghe, Ramal; Liu, Tiebin; Khoury, Muin J.

doi:10.1038/ejhg.2009.209

Cited by 26 publications

(29 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to make the results from both methods comparable, we provide in tables 8 and 9 the our AUC-based approach has the major advantage that it does not depend on a specific arbitrary cut-off value but implicitly incorporates all possible cut-off values into a single measure of accuracy. The use of the AUC is especially appealing after the recent increasing interest in this measure in the molecular and genetic epidemiology field [24,[27][28][29][30][31][32][33] . Wray et al [21] related the maximum value of the AUC of a genetic risk predictor model with the heritability and prevalence of the disease.…”

Section: Nonlinear Effectsmentioning

confidence: 99%

AUC-RF: A New Strategy for Genomic Profiling with Random Forest

et al. 2011

View full text Add to dashboard Cite

Objective: Genomic profiling, the use of genetic variants at multiple loci simultaneously for the prediction of disease risk, requires the selection of a set of genetic variants that best predicts disease status. The goal of this work was to provide a new selection algorithm for genomic profiling. Methods: We propose a new algorithm for genomic profiling based on optimizing the area under the receiver operating characteristic curve (AUC) of the random forest (RF). The proposed strategy implements a backward elimination process based on the initial ranking of variables. Results and Conclusions: We demonstrate the advantage of using the AUC instead of the classification error as a measure of predictive accuracy of RF. In particular, we show that the use of the classification error is especially inappropriate when dealing with unbalanced data sets. The new procedure for variable selection and prediction, namely AUC-RF, is illustrated with data from a bladder cancer study and also with simulated data. The algorithm is publicly available as an R package, named AUCRF, at http://cran.r-project.org/.

show abstract

Section: Nonlinear Effectsmentioning

confidence: 99%

AUC-RF: A New Strategy for Genomic Profiling with Random Forest

et al. 2011

View full text Add to dashboard Cite

show abstract

“…Of the modeling studies, some explicitly obtain weighted risks scores, 5,6,8 whereas others consider different effect sizes for risk alleles in other ways. 4,7 In conclusion, the five most commonly used methods for quantifying the AUC of genetic risk prediction models have similar assumptions, but differ with regard to the input parameters required and the AUC values estimated. The simulation methods yielded consistent AUC estimates and both accurately replicated published empirical AUC values.…”

Section: Discussionmentioning

confidence: 99%

“…[4][5][6][7][8] The methods are referred in this paper by the name of the first author. Three methods use analytical formulas and two use simulations to obtain the AUC.…”

Section: Analytical and Simulation Methodsmentioning

confidence: 99%

“…The AUC values are subsequently obtained from the distribution of likelihood ratios in cases and controls. Second, the analytic method by Moonesinghe 4 obtains the AUC using a formula that requires RRs and frequencies for dominant or recessive effects of the variants. This method approximates the distributions of the number of risk genotypes for cases and controls by normal distributions that are subsequently used to obtain the AUC value.…”

Section: Analytical and Simulation Methodsmentioning

confidence: 99%

“…[4][5][6][7][8] These studies have demonstrated that hundreds of genetic variants are required to obtain an AUC of 0.70 when their effect sizes are small (odds ratio (OR) o1.2), 5 and that the upper limit of the AUC is determined by the heritability of the disease and the population disease risk. 5,9 For example, when the heritability of the disease is 10% and the population disease risk is 20%, the maximum AUC value that can be obtained by genetic risk models will be around 0.80.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Analytical and simulation methods for estimating the potential predictive ability of genetic profiling: a comparison of methods and results

2012

View full text Add to dashboard Cite

Various modeling methods have been proposed to estimate the potential predictive ability of polygenic risk variants that predispose to various common diseases. However, it is unknown whether differences between them affect their conclusions on predictive ability. We reviewed input parameters, assumptions and output of the five most common methods and compared their estimates of the area under the receiver operating characteristic (ROC) curve (AUC) using hypothetical data representing effect sizes and frequencies of genetic variants, population disease risk and number of variants. To assess the accuracy of the estimated AUCs, we aimed to reproduce the AUCs of published empirical studies. All methods assumed that the combined effect of genetic variants on disease risk followed a multiplicative risk model of independent genetic effects, but they either assumed per allele, per genotype or dominant/recessive effects for the genetic variants. Modeling strategy and input parameters differed. Methods used simulation analysis or analytical formulas with effect sizes quantified by odds ratios (ORs) or relative risks. Estimated AUC values were similar for lower ORs (o1.2). When AUCs were larger (40.7) due to variants with strong effects, differences in estimated AUCs between methods increased. The simulation methods accurately reproduced the AUC values of empirical studies, but the analytical methods did not. We conclude that despite differences in input parameters, the modeling methods estimate similar AUC for realistic values of the ORs. When one or more variants have stronger effects and AUC values are higher, the simulation methods tend to be more accurate.

show abstract

Strategies for Developing Prediction Models From Genome‐Wide Association Studies

Pfeiffer

Gail

2013

Genetic Epidemiology

View full text Add to dashboard Cite

Genome-wide association studies (GWASs) have identified hundreds of single nucleotide polymorphisms (SNPs) associated with complex human diseases. However, risk prediction models based on them have limited discriminatory accuracy. It has been suggested that including many such SNPs can improve predictive performance. Here, we studied various aspects of model building to improve discriminatory accuracy, as measured by the area under the receiver operating characteristic curve (AUC), including: (1) How well does a one-phase procedure that selects SNPs and estimates odds ratios on the same data perform? (2) How should training data be allocated between SNP selection (Phase 1) and estimation (Phase 2) in a two-phase procedure? (3) Should SNP selection be based on P-value thresholding or ranking P-values? (4) How many SNPs should be selected? and (5) Is multivariate estimation preferred to univariate estimation in the presence of linkage disequilibrium (LD)? We used realistic estimates of the distributions of genetic effect sizes, allele frequencies, and LD patterns based on GWAS data for Crohn's disease and prostate cancer. Theory and simulations were used to estimate AUC. Empirical risk models based on 10,000 cases and controls had considerably lower AUC than theoretically achievable. The most critical aspect of prediction model building was initial SNP selection. The single-phase procedure achieved higher AUC than the two-phase procedure. Multivariate estimation did not perform as well as univariate (marginal) estimation. For complex diseases and samples of 10,000 or fewer cases and controls, one should limit the number of SNPs to tens or hundreds.

show abstract

Evaluation of the discriminative accuracy of genomic profiling in the prediction of common complex diseases

Cited by 26 publications

References 21 publications

AUC-RF: A New Strategy for Genomic Profiling with Random Forest

AUC-RF: A New Strategy for Genomic Profiling with Random Forest

Analytical and simulation methods for estimating the potential predictive ability of genetic profiling: a comparison of methods and results

Strategies for Developing Prediction Models From Genome‐Wide Association Studies

Contact Info

Product

Resources

About