The Mindful Manager: Validation of a Rounding Leadership Instrument for Residents

Abstract

BACKGROUND: In the context of inpatient general medicine, "rounding" refers to the process of seeing, assessing, and caring for patients as a team. The clinical leadership skills required of residents to lead rounds are essential to inpatient care and clinical education. Assessment of these skills has relevance to developing competent physicians; however, there is an absence of widely accepted tools to specifically measure this competency. OBJECTIVE: To develop and collect validity evidence for a direct observation instrument of internal medicine residents' leadership skills during daily inpatient care rounds for future formative assessment. DESIGN: Prospective observational study. PARTICIPANTS: PGY2 and PGY3 internal medicine residents. MAIN MEASURES: The authors collected inferences of validity evidence according to Kane's validity model. They performed direct observations of PGY2 and PGY3 residents by individual faculty and trained raters and measured inter-rater reliability, using the kappa statistic. Mixed linear regression models were used to compare PGY2 and PGY3 residents. Surveys captured faculty perceptions about value of the instrument. KEY RESULTS: A total of 223 observations were performed in 92 unique individuals. Twenty-four faculty used the observation instrument, of which 18 (75%) completed the post-survey, and 100% agreed that the instrument represented the resident's global leadership abilities. Inter-rater reliability was strong, with an overall kappa statistic equaling 0.82. The mean performance for PGY2 and PGY3 residents was 15.9 (SD 5.1) and 17.7 (SD 4.1), respectively. Adjusting for repeated measures, there was no statistically significant difference between groups. CONCLUSIONS: The authors reported evidence for all four stages of validity and use of the instrument in clinical practice. Their work provides a codification of best practices of rounding leadership, which directly impacts the education of trainees, care of hospitalized patients, and use for formative assessment. The instrument also has the potential to be used for summative assessment.

Designing a Checklist for Directly Observing Use of One-Minute Preceptor Steps on Inpatient Rounds: A Pilot Study

“…7,8 These studies pointed out the educational significance of these tools, despite a few individual items having lower inter-rater reliability. 7,8…”

“…7,8 These studies pointed out the educational significance of these tools, despite a few individual items having lower inter-rater reliability. 7,8 Although our tool demonstrated moderate to substantial agreement, we must evaluate how to improve reliability for our instrument. Ricotta et al 8 attributed their strong inter-rater reliability to iterative construct development and multifaceted faculty development.…”

“…Although our tool demonstrated moderate to substantial agreement, we must evaluate how to improve reliability for our instrument. Ricotta et al 8 attributed their strong inter-rater reliability to iterative construct development and multifaceted faculty development. We propose that additional training for assessors is a key measure to achieve kappa > 0.8.…”

“…Furthermore, designing an easy-to-use binary checklist allowed us to use nonphysician observers, which has previously been performed in the literature. 7,8 The use of nonphysician observers increases the feasibility of applying this tool at other institutions, with the data used as direct feedback to learners and as overall feedback to program leadership.…”

“…Notably, previous studies have shown high kappa values for overall assessment tool but lower kappa values for individual steps. 7,8 These studies pointed out the educational significance of these tools, despite a few individual items having lower inter-rater reliability. 7,8 Although our tool demonstrated moderate to substantial agreement, we must evaluate how to improve reliability for our instrument.…”