Use of the Quality Improvement (QI) Knowledge Application Tool in Assessing Pediatric Resident QI Education

Abstract

Background Assessing the effectiveness of quality improvement curricula is important to improving this area of resident education. Objective To assess the ability of the Quality Improvement Knowledge Application Tool (QIKAT) to differentiate between residents who were provided instruction in QI and those who were not, when scored by individuals not involved in designing the QIKAT, its scoring rubric, or QI curriculum instruction. Methods The QIKAT and a 9-item self-assessment of QI proficiency were administered to an intervention and a control group. The intervention was a longitudinal curriculum consisting of 8 hours of didactic QI training and 6 workshops providing just-in-time training for resident QI projects. Two uninvolved faculty scored the QIKAT. Results A total of 33 residents in the intervention group and 27 in the control group completed the baseline and postcurriculum QIKAT and self-assessment. QIKAT mean intervention group scores were significantly higher than mean control group scores postcurriculum (P < .001). Absolute QIKAT differences were small (of 15 points, intervention group improved from a mean score of 12.8 to 13.2). Interrater agreement as measured by kappa test was low (0.09). Baseline self-assessment showed no differences, and after instruction, the intervention group felt more proficient in QI knowledge than controls in 4 of 9 domains tested. Conclusions The QIKAT detected a statistically significant improvement postintervention, but the absolute differences were small. Self-reported gain in QI knowledge and proficiency agreed with the results of the QIKAT. However, QIKAT limitations include poor interrater agreement and a scoring rubric that lacks specificity. Programs considering using QIKAT to assess curricula should understand these limitations.

Re-thinking How We Teach Quality Improvement: Adding Meaning to an ACGME Requirement

“…Several studies have used objective scoring measures in combination with self-assessment surveys to demonstrate curriculum success. 14,15,[17][18][19][20][21] To our knowledge, only two studies 20,21 have used the QIKAT-R, which was developed to address subjectivity and inconsistent reliability noted in the original version. 8 Our results show a larger improvement from pre-to post-curriculum scores with a difference of 9.6, compared with 2 and 3.3 in comparative studies; however, the mean post-curriculum score of 16.6 falls in-between these studies (15.3 and 19.1).…”

A Project-Based, Resident-Led Quality Improvement Curriculum Within a Pediatric Continuity Clinic

“…To assess residents' ability to apply QI principles to hypothetical clinical scenarios, interns completed a version of the QIKAT that had been adapted for pediatrics and validated in a prior study. 23 The QIKAT provided a clinical scenario and asked the learner to write an aim to target for improvement, a measurement, and a proposed test of change, with up to 2 points awarded for the aim, 1 point for the measurement, 1 point for the test of change, and 1 point for relatedness of the above answers, for a maximum of 5 points per scenario. Residents completed three scenarios as a pretest before the lecture and three scenarios afterward as a posttest.…”

Validity and Reliability of a Tool to Assess Quality Improvement Knowledge and Skills in Pediatrics Residents

“…Glissmeyer et al 15 previously described the development of pediatrics cases adapted from the QI Knowledge Assessment Tool (QIKAT). However, use of the QIKAT questions with pediatrics cases resulted in low interrater reliability, and did not discriminate well between learners with greater and lower QI knowledge.…”

“…However, use of the QIKAT questions with pediatrics cases resulted in low interrater reliability, and did not discriminate well between learners with greater and lower QI knowledge. 15 We designed a new question set that, together with cases developed by Glissmeyer et al, 15 Once the 9 AQIKS questions were selected, we developed a scoring rubric. The point total assigned to each question reflects the complexity of the concept or skill tested.…”

“…Drawbacks of the existing approaches to QI assessment include reliance on self-reports, 6,7 requirement for faculty training or expertise in QI, 8,9 evaluation of only a limited subset of skills necessary to engage in QI, 10,11 and limited validity evidence of instruments. [12][13][14][15] Establishing strategies to measure QI skills and knowledge can help ensure that residency training programs prepare physicians to participate in and lead QI efforts. In pursuit of this goal, experts have called for more robust assessment strategies for QI curricula.…”