Implementation outcome instruments for use in physical healthcare settings: a systematic review

Khadjesari, Zarnie; Boufkhed, Sabah; Vitoratou, Silia; Schatte, Laura; Ziemann, Alexandra; Daskalopoulou, Christina; Uglik‐Marucha, Nora; Sevdalis, Nick; Hull, Louise

doi:10.1186/s13012-020-01027-6

Cited by 49 publications

(45 citation statements)

References 100 publications

(240 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, given the strength of SERP as a simple and interpretable scoring tool, further assessments could be performed to evaluate the more intangible aspects of score implementation. 35 , 36 Such measures would include determining SERP’s long-term sustainability, overall cost-effectiveness, and physician-perceived acceptability. These future assessments might lend credence to SERP as an effective and accurate tool for decision-making within the ED.…”

Section: Discussionmentioning

confidence: 99%

Development and Assessment of an Interpretable Machine Learning Triage Tool for Estimating Mortality After Emergency Admissions

et al. 2021

View full text Add to dashboard Cite

IMPORTANCE Triage in the emergency department (ED) is a complex clinical judgment based on the tacit understanding of the patient's likelihood of survival, availability of medical resources, and local practices. Although a scoring tool could be valuable in risk stratification, currently available scores have demonstrated limitations. OBJECTIVES To develop an interpretable machine learning tool based on a parsimonious list of variables available at ED triage; provide a simple, early, and accurate estimate of patients' risk of death; and evaluate the tool's predictive accuracy compared with several established clinical scores. DESIGN, SETTING, AND PARTICIPANTS This single-site, retrospective cohort study assessed all ED patients between January 1, 2009, and December 31, 2016, who were subsequently admitted to a tertiary hospital in Singapore. The Score for Emergency Risk Prediction (SERP) tool was derived using a machine learning framework. To estimate mortality outcomes after emergency admissions, SERP was compared with several triage systems, including Patient Acuity Category Scale, Modified Early Warning Score, National Early Warning Score, Cardiac Arrest Risk Triage, Rapid Acute Physiology Score, and Rapid Emergency Medicine Score. The initial analyses were completed in October 2020, and additional analyses were conducted in May 2021. MAIN OUTCOMES AND MEASURESThree SERP scores, namely SERP-2d, SERP-7d, and SERP-30d, were developed using the primary outcomes of interest of 2-, 7-, and 30-day mortality, respectively.Secondary outcomes included 3-day mortality and inpatient mortality. The SERP's predictive power was measured using the area under the curve in the receiver operating characteristic analysis. RESULTSThe study included 224 666 ED episodes in the model training cohort (mean [SD] patient age, 63.60 [16.90] years; 113 426 [50.5%] female), 56 167 episodes in the validation cohort (mean [SD] patient age, 63.58 [16.87] years; 28 427 [50.6%] female), and 42 676 episodes in the testing cohort (mean [SD] patient age, 64.85 [16.80] years; 21 556 [50.5%] female). The mortality rates inthe training cohort were 0.8% at 2 days, 2.2% at 7 days, and 5.9% at 30 days. In the testing cohort, the areas under the curve of SERP-30d were 0.821 (95% CI, 0.796-0.847) for 2-day mortality, 0.826 (95% CI, 0.811-0.841) for 7-day mortality, and 0.823 (95% CI, 0.814-0.832) for 30-day mortality and outperformed several benchmark scores. CONCLUSIONS AND RELEVANCEIn this retrospective cohort study, SERP had better prediction performance than existing triage scores while maintaining easy implementation and ease of ascertainment in the ED. It has the potential to be widely applied and validated in different circumstances and health care settings.

show abstract

Section: Discussionmentioning

confidence: 99%

Development and Assessment of an Interpretable Machine Learning Triage Tool for Estimating Mortality After Emergency Admissions

et al. 2021

View full text Add to dashboard Cite

show abstract

“…For the development of the scale and for establishing the analysis plan, we followed the consensus-based standards for the selection of health measurement instruments (COS-MIN) [27,28] and the contemporary psychometrics checklist (ConPsy) [29] recommendations. The scale was developed over four waves between January 2019 and March 2020 (Figure 1).…”

Section: Methodsmentioning

confidence: 99%

Listening to People with Misophonia: Exploring the Multiple Dimensions of Sound Intolerance Using a New Psychometric Tool, the S-Five, in a Large Sample of Individuals Identifying with the Condition

Vitoratou

Uglik‐Marucha

Hayes

et al. 2021

Psych

Self Cite

105

View full text Add to dashboard Cite

Misophonia involves a strong emotional response to certain sounds and can cause significant distress and functional impairment. The aim of this study was to present and psychometrically evaluate a new, multidimensional measure of misophonia, the S-Five. The study also aimed to present and test a supplementary tool, a checklist of triggers that measure the nature and intensity of reactions. The stages of development for the measure are described. Psychometric testing on the final version of the tool was conducted using a sample of 828 individuals who identified with having misophonia. Analyses included factor structure assessment, measurement invariance testing, reliability (test–retest and internal consistency), and (concurrent) convergent validity assessment. Five factors emerged in the S-Five as dimensions of the experience of misophonia: internalising appraisals, externalising appraisals, sense of emotional threat, outbursts, and impact. No measurement bias was identified with respect to gender and age. All reliability and validity indices were satisfactory. The S-Five is a multidimensional measurement scale with satisfactory psychometric properties and will be a valuable tool for improving understanding of misophonia in research and clinical settings.

show abstract

“…Given the relatively good prediction performance and advantages of SERP, prospective studies will be designed to further validate its performance for implementation into real-life practice. Also, given the strengths of SERP as a simple yet understandable scoring tool, further assessments must be carried out to evaluate the more intangible aspects of score implementation 34,35 . Such measures would include a determination of SERP’s long-term sustainability, overall cost-effectiveness, and most importantly, physician perceived acceptability and score take-up rates.…”

Section: Discussionmentioning

confidence: 99%

Score for Emergency Risk Prediction (SERP): An Interpretable Machine Learning AutoScore–Derived Triage Tool for Predicting Mortality after Emergency Admissions

Ong

Jnmh

et al. 2021

Preprint

View full text Add to dashboard Cite

Importance: Triage in the emergency department (ED) for admission and appropriate level of hospital care is a complex clinical judgement based on tacit understanding of the patient's likely acute course, availability of medical resources, and local practices. While a scoring tool could be valuable in triage, currently available tools have demonstrated limitations. Objective: To develop a tool based on a parsimonious list of predictors available early at ED triage, to provide a simple, early, and accurate estimate of short-term mortality risk, the Score for Emergency Risk Prediction (SERP), and evaluate its predictive accuracy relative to published tools. Design, Setting, and Participants: We performed a single-site, retrospective study for all emergency department (ED) patients between January 2009 and December 2016 admitted in a tertiary hospital in Singapore. SERP was derived using the machine learning framework for developing predictive models, AutoScore, based on six variables easily available early in the ED care process. Using internal validation, the SERP was compared to the current triage system, Patient Acuity Category Scale (PACS), Modified Early Warning Score (MEWS), National Early Warning Score (NEWS), Cardiac Arrest Risk Triage (CART), and Charlson Comorbidity Index (CCI) in predicting both primary and secondary outcomes in the study. Main Outcomes and Measures: The primary outcome of interest was 30-day mortality. Secondary outcomes include 2-day mortality, inpatient mortality, 30-day post-discharge mortality and 1-year mortality. The SERP's predictive power was measured using the area under the curve (AUC) in the receiver operating characteristic (ROC) analysis. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were calculated under the optimal threshold, defined as the point nearest to the upper-left corner of the ROC curve. Results: We included 224,666 ED episodes in the model training cohort, 56,167 episodes in the validation cohort and 42,676 episodes in the testing cohort. 18,797 (5.8%) of them died in 30 days after their ED visits. Evaluated on the testing set, SERP outperformed several benchmark scores in predicting 30-day mortality and other mortality-related outcomes. Under cut-off score of 27, SERP achieved a sensitivity of 72.6% (95% confidence interval [CI]: 70.7-74.3%), a specificity of 77.8% (95% CI: 77.5-78.2), a positive predictive value of 15.8% (15.4-16.2%) and a negative predictive value of 98% (97.9-98.1%). Conclusions: SERP showed better prediction performance than existing triage scores while maintaining easy implementation and ease of ascertainment at the ED. It has the potential to be widely applied and validated in different circumstances and healthcare settings.

show abstract

Implementation outcome instruments for use in physical healthcare settings: a systematic review

Cited by 49 publications

References 100 publications

Development and Assessment of an Interpretable Machine Learning Triage Tool for Estimating Mortality After Emergency Admissions

Development and Assessment of an Interpretable Machine Learning Triage Tool for Estimating Mortality After Emergency Admissions

Listening to People with Misophonia: Exploring the Multiple Dimensions of Sound Intolerance Using a New Psychometric Tool, the S-Five, in a Large Sample of Individuals Identifying with the Condition

Score for Emergency Risk Prediction (SERP): An Interpretable Machine Learning AutoScore–Derived Triage Tool for Predicting Mortality after Emergency Admissions

Contact Info

Product

Resources

About