Validating and Updating GRASP: An Evidence-Based Framework for Grading and Assessment of Clinical Predictive Tools

Abstract

Background: When selecting predictive tools, for implementation in their clinical practice or for recommendation in clinical guidelines, clinicians are challenged with an overwhelming and ever-growing number of tools. Many of these have never been implemented or evaluated for comparative effectiveness. To overcome this challenge, the authors developed an evidence-based framework for grading and assessment of predictive tools (GRASP), based on the critical appraisal of their published evidence. The objective of this study is to validate, update GRASP, and evaluate its reliability. Methods:The study is composed of two parts. The first includes validating and updating the GRASP framework and the second includes evaluating the framework reliability. For the first part, an online survey was developed to collect the responses of a wide international group of experts; identified as healthcare researchers who have published studies on developing, implementing or evaluating predictive tools and clinical decision support systems. For the second part, the interrater reliability of the framework, to assign grades to eight predictive tools by two independent users, will be evaluated.Results: Out of 882 invited experts, 81 valid responses were received. On a fivepoints Likert scale, experts overall strongly agreed to GRASP evaluation criteria of predictive tools (4.35/5). Experts strongly agreed to six criteria; predictive performance (4.87/5) and predictive performance levels (4.44/5), usability (4.68/5) and potential 2 effect (4.61/5), post-implementation impact (4.78/5) and evidence direction (4.26/5).Experts somewhat agreed to one criterion; post-implementation impact levels (4.16/5).Experts were neutral about one criterion; usability is higher than potential effect (2.97/5). Two thirds of the experts provided recommendations to six open-ended questions regarding adding, removing or changing evaluation criteria. Over half of the experts suggested that the potential effect, as an evaluation criterion, should be higher than the usability. Experts highlighted the importance of reporting the quality of studies and the strength of evidence supporting the grades assigned to predictive tools. The GRASP concept and its detailed report were updated based on experts' feedback.Following validation, the interrater reliability of the GRASP framework, to produce accurate and consistent results by two independent users, was tested and the framework found to be reliable. Answering open-ended questions, the two independent users reported the GRASP framework was logical, useful, and easy to use. Discussion and Conclusion:The GRASP framework grades predictive tools based on the critical appraisal of the published evidence across three dimensions: 1) Phase of evaluation; 2) Level of evidence; and 3) Direction of evidence. The final grade of a tool is based on the highest phase of evaluation, supported by the highest level of positive evidence, or mixed evidence that supports positive conclusion. GRASP is not meant to be prescriptive; it provi...