Development and Validation of HealthImpact: An Incident Diabetes Prediction Model Based on Administrative Data

McCoy, Rozalina G.; Nori, Vijay S.; Smith, Steven A.; Hane, Christopher A.

doi:10.1111/1475-6773.12461

Cited by 11 publications

(22 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The ADRD predictive model was trained and tested using a nested case-control study design [8]. Step-wise demonstration of how the study population was assembled is depicted in Fig 2.…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Identifying incident dementia by applying machine learning to a very large administrative claims dataset

et al. 2019

Self Cite

View full text Add to dashboard Cite

Alzheimer's disease and related dementias (ADRD) are highly prevalent conditions, and prior efforts to develop predictive models have relied on demographic and clinical risk factors using traditional logistical regression methods. We hypothesized that machine-learning algorithms using administrative claims data may represent a novel approach to predicting ADRD. Using a national de-identified dataset of more than 125 million patients including over 10,000 clinical, pharmaceutical, and demographic variables, we developed a cohort to train a machine learning model to predict ADRD 4–5 years in advance. The Lasso algorithm selected a 50-variable model with an area under the curve (AUC) of 0.693. Top diagnosis codes in the model were memory loss (780.93), Parkinson’s disease (332.0), mild cognitive impairment (331.83) and bipolar disorder (296.80), and top pharmacy codes were psychoactive drugs. Machine learning algorithms can rapidly develop predictive models for ADRD with massive datasets, without requiring hypothesis-driven feature engineering.

show abstract

“…The ADRD predictive model was trained and tested using a nested case-control study design [8]. Step-wise demonstration of how the study population was assembled is depicted in Fig 2.…”

Section: Methodsmentioning

confidence: 99%

“…with a cStatistic of 0.9 and 0.76, respectively. McCoy, et al [8] developed a predictive model trained on medical and pharmacy claims for 473,049 people to identify those at risk for type 2 diabetes. The model with 48 variables had a cStatistic of 0.808.…”

Section: Introductionmentioning

confidence: 99%

Identifying incident dementia by applying machine learning to a very large administrative claims dataset

et al. 2019

Self Cite

View full text Add to dashboard Cite

show abstract

“…We selected two published patient-level prediction models that used a case-control design to develop the models using observational healthcare data. The rst predicted future Alzheimer's risk 6 and the second predicted future type 2 diabetes risk [7]. We replicated the two case-control models by following the published process, but because we do not have access to the same patient-level data, we instead use the Optum® De-Identi ed Clinformatics® Data Mart Database -Socio-Economic Status (Optum Claims), a US claims database.…”

Section: Replication Of Case-control Patient-level Predictions and Comentioning

confidence: 99%

“…The two most widely implemented study designs for extracting labelled data from observational databases are the cohort design [5] and the case-control design [6,7]. Figure 1 illustrates the differences between the designs.…”

Section: Introductionmentioning

confidence: 99%

Design Matters in Patient-Level Prediction: Evaluation of a Cohort vs. Case-Control Design When Developing Predictive Models in Observational Healthcare Datasets

Reps

Ryan

Schuemie

2021

Preprint

View full text Add to dashboard Cite

Background: The study design used to develop prediction models in observational healthcare databases (e.g., case-control and cohort) may impact the clinical usefulness. We aim to quantify how the choice of design impacts prediction model performance. Aim: To empirically investigate differences between models developed using a case-control design and a cohort design.Methods: Using a US claims database, we replicated two published prediction models (dementia and type 2 diabetes) which were developed using a case-control design, and also train models for the same prediction questions using cohort designs. We validated each model on data mimicking the point in time the models would be applied in clinical practice. We calculate the models’ discrimination and calibration-in-the-large performances.Results: The dementia models obtained area under the receiver operating characteristics of 0.560 and 0.897 for the case-control and cohort designs respectively. The type 2 diabetes models obtained area under the receiver operating characteristics of 0.733 and 0.727 for the case-control and cohort designs respectively. The dementia and diabetes case-control models were both poorly calibrated, whereas the dementia cohort model achieved good calibration. We show that careful construction of a case-control design can lead to comparable discriminative performance as a cohort design, but case-control designs generally oversample the outcome leading to miscalibration. Conclusion: Any case-control design can be converted to a cohort design. We recommend that researchers with observational data use the less subjective and generally better calibrated cohort design. However, if a carefully constructed case-control design is used, then the model must be prospectively validated using a cohort design for fair evaluation and be recalibrated.

show abstract

“…However, this article mentions using all medical records within 10 years prior to the index date which is not available in most systems or settings. While the body of work using ML models in this area is limited, they have been used extensively for identifying people at risk in a variety of other areas such as diabetes [13], chronic kidney disease [14], heart failure [15], cardiovascular disease [16], etc. The advantage of these methodologies is the ability to choose the most predictive out of tens of thousands of features without the need for clinical input, and create models which typically have better (or at least the same) accuracy than the curated models.…”

Section: Introductionmentioning

confidence: 99%

Deep neural network models for identifying incident dementia using claims and EHR datasets

et al. 2020

Self Cite

View full text Add to dashboard Cite

This study investigates the use of deep learning methods to improve the accuracy of a predictive model for dementia, and compares the performance to a traditional machine learning model. With sufficient accuracy the model can be deployed as a first round screening tool for clinical follow-up including neurological examination, neuropsychological testing, imaging and recruitment to clinical trials. Seven cohorts with two years of data, three to eight years prior to index date, and an incident cohort were created. Four trained models for each cohort, boosted trees, feed forward network, recurrent neural network and recurrent neural network with pre-trained weights, were constructed and their performance compared using validation and test data. The incident model had an AUC of 94.4% and F1 score of 54.1%. Eight years removed from index date the AUC and F1 scores were 80.7% and 25.6%, respectively. The results for the remaining cohorts were between these ranges. Deep learning models can result in significant improvement in performance but come at a cost in terms of run times and hardware requirements. The results of the model at index date indicate that this modeling can be effective at stratifying patients at risk of dementia. At this time, the inability to sustain this quality at longer lead times is more an issue of data availability and quality rather than one of algorithm choices.

show abstract

Development and Validation of HealthImpact: An Incident Diabetes Prediction Model Based on Administrative Data

Abstract: HealthImpact is an efficient and effective method of risk stratification for incident diabetes that is not predicated on patient-provided information or laboratory tests.

Cited by 11 publications

References 33 publications

Identifying incident dementia by applying machine learning to a very large administrative claims dataset

Identifying incident dementia by applying machine learning to a very large administrative claims dataset

Design Matters in Patient-Level Prediction: Evaluation of a Cohort vs. Case-Control Design When Developing Predictive Models in Observational Healthcare Datasets

Deep neural network models for identifying incident dementia using claims and EHR datasets

Contact Info

Product

Resources

About