53. Development of Progress Note Assessment and Plan Evaluation (Pnape) Tool

Abstract

BACKGROUND: Quality improvement (QI) education is a requirement at all stages of medical education training. Pediatric training programs are required to assess their trainees on quality improvement reflection and knowledge as part of the Pediatric Milestones. However, there is currently no validated tool for use in pediatrics to assess reflection on quality improvement opportunities. The Mayo Evaluation of Reflection on Improvement Tool (MERIT) has been previously published with validity evidence for use in internal medicine programs, but it has not been studied for use in pediatric training programs.OBJECTIVE: To validate the MERIT assessment tool for use in pediatric residency programs to aid in quality improvement milestone evaluation. DESIGN/METHODS: All University of Texas Southwestern (UTSW) pediatric residents completed annual improvement reflections between 2015-2017. De-identified reflections were assessed by five independent reviewers. Reviewers were oriented to the MERIT tool and compared 10 initial evaluations to ensure consistency. Completed MERIT evaluation scores were compared. Inter-rater reliability for each item on the tool was compared with intra-class correlation coefficients (ICC) and corresponding 95% confidence intervals (CI) using a mean-rating, absolute-agreement, 2-way mixed effects model. Internal consistency for each section and the overall tool was evaluated with Cronbach's alpha and inter-item correlations. All calculations were done with SPSS statistical package version 25.RESULTS: 150 resident reflections were evaluated yielding 749 evaluation forms for study. Item mean scores were highest for the Problem of Merit section (3.19) and lowest for the Reflection on System Characteristics of Quality Improvement (1.94). Interrater reliability was good for all 18 items on the MERIT tool (ICC range: 0.78 -0.90). Inter-rater reliability was good to excellent for 13 of the 18 items on the tool when 95% CI was used. Internal consistency was excellent (Cronbach's alpha 0.93 overall and 0.94 -0.95 for each section). The Cronbach's alpha did not improve if any single item was removed. Inter-item correlations were high for all three sections, but highest for the Problem of Merit section (0.67 -0.92).CONCLUSION(S): Validity evidence supports the use of the MERIT in pediatric residency programs to assess resident reflections on quality improvement. The pediatric validity evidence is consistent with the previously published validation study in an Internal Medicine residency program.

Resident Workshop to Improve Inpatient Documentation Using the Progress Note Assessment and Plan Evaluation (PNAPE) Tool

“…10 In a forthcoming validation study, we found an interrater reliability of κ = .89 (95% confidence interval: .68-.97) and a moderate to strong positive association between mean PNAPE and PDQI-9 scores (Pearson's r = .63, p = .027). 11 …”