A tutorial on calibration measurements and calibration models for clinical prediction models

Huang, Yingxiang; Li, Wentao; Macheret, Fima; Gabriel, Rodney A.; Ohno-Machado, Lucila

doi:10.1093/jamia/ocz228

Cited by 212 publications

(204 citation statements)

References 45 publications

Supporting

Mentioning

200

Contrasting

Unclassified

Order By: Relevance

“…We tested discrimination with area under the receiver operating characteristics curve (AUC) and calibration with Matthews correlation coefficient. A receiver operator characteristic (ROC) curve plots the true positive rate on y -axis against the false positive rate on x -axis ( Huang et al, 2020 ). AUC is score that measures the area under the ROC curve and it ranges from 0.50 to 1.0 with higher values meaning higher discrimination.…”

Section: Methodsmentioning

confidence: 99%

Predicting CoVID-19 community mortality risk using machine learning and development of an online prognostic tool

Das

Mishra²,

Gopalan

2020

PeerJ

View full text Add to dashboard Cite

Background The recent pandemic of CoVID-19 has emerged as a threat to global health security. There are very few prognostic models on CoVID-19 using machine learning. Objectives To predict mortality among confirmed CoVID-19 patients in South Korea using machine learning and deploy the best performing algorithm as an open-source online prediction tool for decision-making. Materials and Methods Mortality for confirmed CoVID-19 patients (n = 3,524) between January 20, 2020 and May 30, 2020 was predicted using five machine learning algorithms (logistic regression, support vector machine, K nearest neighbor, random forest and gradient boosting). The performance of the algorithms was compared, and the best performing algorithm was deployed as an online prediction tool. Results The logistic regression algorithm was the best performer in terms of discrimination (area under ROC curve = 0.830), calibration (Matthews Correlation Coefficient = 0.433; Brier Score = 0.036) and. The best performing algorithm (logistic regression) was deployed as the online CoVID-19 Community Mortality Risk Prediction tool named CoCoMoRP (https://ashis-das.shinyapps.io/CoCoMoRP/). Conclusions We describe the development and deployment of an open-source machine learning tool to predict mortality risk among CoVID-19 confirmed patients using publicly available surveillance data. This tool can be utilized by potential stakeholders such as health providers and policymakers to triage patients at the community level in addition to other approaches.

show abstract

Section: Methodsmentioning

confidence: 99%

Predicting CoVID-19 community mortality risk using machine learning and development of an online prognostic tool

Das

Mishra²,

Gopalan

2020

PeerJ

View full text Add to dashboard Cite

show abstract

“…Brier score simultaneously account for discrimnation and calibration. [15] A smaller Brier score indicates better performance. In addition, the gradient boosting algorithm was used to estimate the relative contributions of the predictors and draw the variable importance plot.…”

Section: Evaluation Of the Performance Of The Algorithmsmentioning

confidence: 99%

Predicting community mortality risk due to CoVID-19 using machine learning and development of a prediction tool

Das

Mishra²,

Gopalan

2020

Preprint

View full text Add to dashboard Cite

Background: The recent pandemic of CoVID-19 has emerged as a threat to global health security. There are a very few prognostic models on CoVID-19 using machine learning.Objectives: To predict mortality among confirmed CoVID-19 patients in South Korea using machine learning and deploy the best performing algorithm as an open-source online prediction tool for decision-making. 299) between January 20, 2020 and April 30, 2020 was predicted using five machine learning algorithms (logistic regression, support vector machine, K nearest neighbor, random forest and gradient boosting). Performance of the algorithms was compared, and the best performing algorithm was deployed as an online prediction tool. Materials and methods: Mortality for confirmedResults: The random forest algorithm was the best performer in terms of predictive ability (accuracy=0.981), discrimination (area under ROC curve=0.886), calibration (Matthews Correlation Coefficient=0.459; Brier Score=0.063) and. The best performer algorithm (random forest) was deployed as the online CoVID-19 Community Mortality Risk Prediction tool named CoCoMoRP (https://ashis-das.shinyapps.io/CoCoMoRP/). Conclusions:We describe the development and deployment of an open-source machine learning tool to predict mortality risk among CoVID-19 confirmed patients using publicly available surveillance data. This tool can be utilized by potential stakeholders such as health providers and policy makers to triage patients at the community level in addition to other approaches.

show abstract

“…A receiver operator characteristic (ROC) curve plots the true positive rate on y-axis against the false positive rate on x-axis. 17 AUC is score that measures the area under the ROC curve and it ranges from 0.50 to 1.0 with higher values meaning higher discrimination. Matthews correlation coefficient (MCC) is a measure that takes into account all four predictive classes -true positive, true negative, false positive and false negative.…”

Section: Manuscript To Be Reviewedmentioning

confidence: 99%

“…18 Brier score simultaneously account for discrimnation and calibration. 17 A smaller Brier score indicates better performance. We also estimated accuracy, sensitivity and specificity.…”

Section: Manuscript To Be Reviewedmentioning

confidence: 99%

See 1 more Smart Citation

Peer Review #1 of "Predicting CoVID-19 community mortality risk using machine learning and development of an online prognostic tool (v0.2)"

Jaffé¹

2020

View full text Add to dashboard Cite

Background. The recent pandemic of CoVID-19 has emerged as a threat to global health security. There are very few prognostic models on CoVID-19 using machine learning. Objectives. To predict mortality among confirmed CoVID-19 patients in South Korea using machine learning and deploy the best performing algorithm as an open-source online prediction tool for decision-making. Materials and methods. Mortality for confirmed CoVID-19 patients (n=3,524) between January 20, 2020 and May 30, 2020 was predicted using five machine learning algorithms (logistic regression, support vector machine, K nearest neighbor, random forest and gradient boosting). The performance of the algorithms was compared, and the best performing algorithm was deployed as an online prediction tool. Results. The logistic regression algorithm was the best performer in terms of discrimination (area under ROC curve=0.830), calibration (Matthews Correlation Coefficient=0.433; Brier Score=0.036) and. The best performing algorithm (logistic regression) was deployed as the online CoVID-19 Community Mortality Risk Prediction tool named CoCoMoRP (https://ashis-das.shinyapps.io/CoCoMoRP/). Conclusions. We describe the development and deployment of an open-source machine learning tool to predict mortality risk among CoVID-19 confirmed patients using publicly available surveillance data. This tool can be utilized by potential stakeholders such as health providers and policymakers to triage patients at the community level in addition to other approaches.

show abstract

A tutorial on calibration measurements and calibration models for clinical prediction models

Cited by 212 publications

References 45 publications

Predicting CoVID-19 community mortality risk using machine learning and development of an online prognostic tool

Predicting CoVID-19 community mortality risk using machine learning and development of an online prognostic tool

Predicting community mortality risk due to CoVID-19 using machine learning and development of a prediction tool

Peer Review #1 of "Predicting CoVID-19 community mortality risk using machine learning and development of an online prognostic tool (v0.2)"

Contact Info

Product

Resources

About