Use of Ecological Momentary Assessment to Guide Curricular Change in Graduate Medical Education

Abstract

Purpose To assess whether a novel evaluation tool could guide curricular change in an internal medicine residency program. Method The authors developed an 8-item Ecological Momentary Assessment tool and collected daily evaluations from residents of the relative educational value of 3 differing ambulatory morning report formats (scale: 8 = best, 0 = worst). From the evaluations, they made a targeted curricular change and used the tool to assess its impact. Results Residents completed 1388 evaluation cards for 223 sessions over 32 months, with a response rate of 75.3%. At baseline, there was a decline in perceived educational value with advancing postgraduate (PGY) year for the overall mean score (PGY-1, 7.4; PGY-2, 7.2; PGY-3, 7.0; P < .01) and for percentage reporting greater than 2 new things learned (PGY-1, 77%; PGY-2, 66%; PGY-3, 50%; P < .001). The authors replaced the format of a lower scoring session with one of higher cognitive content to target upper-level residents. The new session's mean score improved (7.1 to 7.4; P = .03); the adjusted odds ratios before and after the change for percentage answering, “Yes, definitely” to “Area I need to improve” was 2.53 (95% confidence interval [CI], 1.45–4.42; P = .001) and to “Would recommend to others,” it was 2.08 (95% CI, 1.12–3.89; P = .05). Conclusions The Ecological Momentary Assessment tool successfully guided ambulatory morning report curricular changes and confirmed successful curricular impact. Ecological Momentary Assessment concepts of multiple, frequent, timely evaluations can be successfully applied in residency curriculum redesign.

Use of Ecological Momentary Assessment to Determine Which Structural Factors Impact Perceived Teaching Quality of Attending Rounds

“…Ecological Momentary Assessment (EMA) is an evaluation method we used in prior work to reform our ambulatory curriculum. 11 Social scientists developed EMA 12,13 to measure participants' behavior and cognition, with the key principle of obtaining multiple, brief evaluations as immediately after the event as possible. The EMA minimizes recall bias, gives more sensitive assessments, and…”