Advancing Our Understanding of Narrative Comments Generated by Direct Observation Tools: Lessons From the Psychopharmacotherapy-Structured Clinical Observation

Abstract

Background While prior research has focused on the validity of quantitative ratings generated by direct observation tools, much less is known about the written comments. Objective This study examines the quality of written comments and their relationship with checklist scores generated by a direct observation tool, the Psychopharmacotherapy-Structured Clinical Observation (P-SCO). Methods From 2008 to 2012, faculty in a postgraduate year 3 psychiatry outpatient clinic completed 601 P-SCOs. Twenty-five percent were randomly selected from each year; the sample included 8 faculty and 57 residents. To assess quality, comments were coded for valence (reinforcing or corrective), behavioral specificity, and content. To assess the relationship between comments and scores, the authors calculated the correlation between comment and checklist score valence and examined the degree to which comments and checklist scores addressed the same content. Results Ninety-one percent of the comments were behaviorally specific. Sixty percent were reinforcing, and 40% were corrective. Eight themes were identified, including 2 constructs not adequately represented by the checklist. Comment and checklist score valence was moderately correlated (Spearman's rho = 0.57, P < .001). Sixty-seven percent of high and low checklist scores were associated with a comment of the same valence and content. Only 50% of overall comments were associated with a checklist score of the same valence and content. Conclusions A direct observation tool such as the P-SCO can generate high-quality written comments. Narrative comments both explain checklist scores and convey unique content. Thematic coding of comments can improve the content validity of a checklist.

A mobile app to capture EPA assessment data: Utilizing the consolidated framework for implementation research to identify enablers and barriers to engagement

“…Prior to the implementation of the EPA app, faculty used a paper-based direct observation tool that included a comprehensive 27-item checklist, an overall EPA rating, and prompts for both reinforcing and corrective comments. This tool had been studied in several settings with evidence for validity and generates, on average, five highly specific comments with a 3:2 ratio of reinforcing to corrective [ 11 , 41 – 43 ]. All faculty agreed to participate.…”

“…This outcome stemmed from the intentional design decision to limit the assessment to a single rating and a single text box in order to maximize efficiency and ease of use. Longer direct observation tools have been shown to generate multiple comments per observation [43,48,58]. We do not know what the optimal number of comments is from a learning and behavior change perspective, but this finding suggests that an overall program of workplace-based assessment may want to include a mix of assessment tools that generate both brief and more detailed comments.…”

Wresting with Implementation: a Step-By-Step Guide to Implementing Entrustable Professional Activities (EPAs) in Psychiatry Residency Programs

“…When aggregated over multiple observations, quantitative data can provide a powerful signal of the direction a trainee is going with respect to competency acquisition. Narrative comments add explanation and guidance about what was done well or not so well that is essential for the learner and CCC to have in order to place the scores in context [14].…”

“…Moreover, the P-SCO has been found to generate highquality narrative comments that are behaviorally specific and balanced between being corrective and re-enforcing. The comments either add unique content or elaborate on the "why" behind a low or high quantitative score [14,15].…”

Workplace-based assessments of entrustable professional activities in a psychiatry core clerkship: an observational study

“…This value depends on the context, content, and quality of the feedback resulting from the WBAs [ 15 – 21 ]. While a number of studies have explored the potential of WBAs for generating high-quality narrative feedback in GME [ 22 – 25 ], little is known about the relationship between WBAs based on an entrustment-scale, their narrative feedback output, and the perceived need for supervision (i.e., self-entrustment) in early-stage clinical students. In particular, changes in self-entrustment can be used as an indicator of self-efficacy [ 26 – 29 ].…”