Evaluating the Impact of the Grading and Assessment of Predictive Tools Framework on Clinicians and Health Care Professionals’ Decisions in Selecting Clinical Predictive Tools: Randomized Controlled Trial

Abstract

<jats:sec> <jats:title>Background</jats:title> <jats:p>While selecting predictive tools for implementation in clinical practice or for recommendation in clinical guidelines, clinicians and health care professionals are challenged with an overwhelming number of tools. Many of these tools have never been implemented or evaluated for comparative effectiveness. To overcome this challenge, the authors developed and validated an evidence-based framework for grading and assessment of predictive tools (the GRASP framework). This framework was based on the critical appraisal of the published evidence on such tools.</jats:p> </jats:sec> <jats:sec> <jats:title>Objective</jats:title> <jats:p>The aim of the study was to examine the impact of using the GRASP framework on clinicians’ and health care professionals’ decisions in selecting clinical predictive tools.</jats:p> </jats:sec> <jats:sec> <jats:title>Methods</jats:title> <jats:p>A controlled experiment was conducted through a web-based survey. Participants were randomized to either review the derivation publications, such as studies describing the development of the predictive tools, on common traumatic brain injury predictive tools (control group) or to review an evidence-based summary, where each tool had been graded and assessed using the GRASP framework (intervention group). Participants in both groups were asked to select the best tool based on the greatest validation or implementation. A wide group of international clinicians and health care professionals were invited to participate in the survey. Task completion time, rate of correct decisions, rate of objective versus subjective decisions, and level of decisional conflict were measured.</jats:p> </jats:sec> <jats:sec> <jats:title>Results</jats:title> <jats:p>We received a total of 194 valid responses. In comparison with not using GRASP, using the framework significantly increased correct decisions by 64%, from 53.7% to 88.1% (88.1/53.7=1.64; t193=8.53; P&lt;.001); increased objective decision making by 32%, from 62% (3.11/5) to 82% (4.10/5; t189=9.24; P&lt;.001); decreased subjective decision making based on guessing by 20%, from 49% (2.48/5) to 39% (1.98/5; t188=−5.47; P&lt;.001); and decreased prior knowledge or experience by 8%, from 71% (3.55/5) to 65% (3.27/5; t187=−2.99; P=.003). Using GRASP significantly decreased decisional conflict and increased the confidence and satisfaction of participants with their decisions by 11%, from 71% (3.55/5) to 79% (3.96/5; t188=4.27; P&lt;.001), and by 13%, from 70% (3.54/5) to 79% (3.99/5; t188=4.89; P&lt;.001), respectively. Using GRASP decreased the task completion time, on the 90th percentile, by 52%, from 12.4 to 6.4 min (t193=−0.87; P=.38). The average System Usability Scale of the GRASP framework was very good: 72.5% and 88% (108/122) of the participants found the GRASP useful.</jats:p> </jats:sec> <jats:sec> <jats:title>Conclusions</jats:title> <jats:p>Using GRASP has positively supported and significantly improved evidence-based decision making. It has increased the accuracy and efficiency of selecting predictive tools. GRASP is not meant to be prescriptive; it represents a high-level approach and an effective, evidence-based, and comprehensive yet simple and feasible method to evaluate, compare, and select clinical predictive tools.</jats:p> </jats:sec>

Evaluating Usability of IRAIVI Pregnancy Prediction Model Using System Usability Scale

“…Whenever researchers and/or professionals require a gauge for perceived usability for any system, it is highly advisable to contemplate employing the SUS. 13,14 An application of precision techniques to predict progression to medical conditions (diabetes mellitus) provides preliminary improvement in prediction. 6,15 Understanding the mechanics of predictive modelling will help in troubleshooting and improving the performance of the model.…”