Measuring Medical Housestaff Teamwork Performance Using Multiple Direct Observation Instruments: Comparing Apples and Apples

Weingart, Saul N.; Yaghi, Omar; Wetherell, Matthew; Sweeney, Megan

doi:10.1097/acm.0000000000002238

Cited by 10 publications

(12 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…30 Another group applied 9 different teamwork observation instruments designed for other settings to the context of daily rounds and found discordant assessments across instruments. 10 They concluded that existing frameworks may not be suitable for daily rounds and recommended the development of novel instruments for rounding leadership, and collecting validity evidence, which we have accomplished here.…”

Section: (100%)mentioning

confidence: 97%

“…Approaches attempting to adapt current leadership assessments to daily rounds have been limited by variability, consistency, and validity. 10 Further, alternative methods often rely on subjective measures and are prone to bias and halo effect, 11,12 while others have not been used in the clinical setting to capture real-life performance. Finally, there is an absence of codified best practices 13 in rounding leadership to serve as guiding frameworks.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The Mindful Manager: Validation of a Rounding Leadership Instrument for Residents

et al. 2020

View full text Add to dashboard Cite

BACKGROUND: In the context of inpatient general medicine, "rounding" refers to the process of seeing, assessing, and caring for patients as a team. The clinical leadership skills required of residents to lead rounds are essential to inpatient care and clinical education. Assessment of these skills has relevance to developing competent physicians; however, there is an absence of widely accepted tools to specifically measure this competency. OBJECTIVE: To develop and collect validity evidence for a direct observation instrument of internal medicine residents' leadership skills during daily inpatient care rounds for future formative assessment. DESIGN: Prospective observational study. PARTICIPANTS: PGY2 and PGY3 internal medicine residents. MAIN MEASURES: The authors collected inferences of validity evidence according to Kane's validity model. They performed direct observations of PGY2 and PGY3 residents by individual faculty and trained raters and measured inter-rater reliability, using the kappa statistic. Mixed linear regression models were used to compare PGY2 and PGY3 residents. Surveys captured faculty perceptions about value of the instrument. KEY RESULTS: A total of 223 observations were performed in 92 unique individuals. Twenty-four faculty used the observation instrument, of which 18 (75%) completed the post-survey, and 100% agreed that the instrument represented the resident's global leadership abilities. Inter-rater reliability was strong, with an overall kappa statistic equaling 0.82. The mean performance for PGY2 and PGY3 residents was 15.9 (SD 5.1) and 17.7 (SD 4.1), respectively. Adjusting for repeated measures, there was no statistically significant difference between groups. CONCLUSIONS: The authors reported evidence for all four stages of validity and use of the instrument in clinical practice. Their work provides a codification of best practices of rounding leadership, which directly impacts the education of trainees, care of hospitalized patients, and use for formative assessment. The instrument also has the potential to be used for summative assessment.

show abstract

Section: (100%)mentioning

confidence: 97%

Section: Introductionmentioning

confidence: 99%

The Mindful Manager: Validation of a Rounding Leadership Instrument for Residents

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Fewer authors appeared to leverage constructivist/interpretivist framings (e.g., Christensen et al, 2018;Pool et al, 2018). We noted instances in which observers were considered objective, but fallible, interchangeable and as contributing error that could be mitigated through training (e.g., Biagioli et al, 2017;Cameron et al, 2017;Dory et al, 2018;Naumann et al, 2016;Park et al, , 2016Park et al, , , 2017Roberts et al, 2017a, b;Turner et al, 2017;Weingart et al, 2018). By contrast, others appeared to value observer subjectivity, and positioned the variation between observers as meaningful (e.g., Chahine et al, 2016;Christensen et al, 2018;Pool et al, 2018).…”

Section: Differences In the Way Assessment Features Are Enacted: Suggesting Variable Positionalitymentioning

confidence: 98%

“…Authors' descriptions on how assessment features were informed by philosophical positions were either vague, unclear, or not reported, and thus required a high degree of interpretation or inference for each feature. (e.g., Biagioli et al, 2017;DeMuth et al, 2018;Gingerich et al, 2017;Ginsburg et al, 2017;Hauer et al, 2018;Li et al, 2017;Martin et al, 2018;Mink et al, 2018;Naidoo et al, 2017;Naumann et al, 2016;Weingart et al, 2018)). This need to interpret led us to call multiple team meetings to discuss and debate the positionality of authors' potential positions.…”

Section: Philosophical Positions As Vague Unclear or Not Reportedmentioning

confidence: 99%

Implicit and inferred: on the philosophical positions informing assessment science

Tavares

Hodwitz

Rowland

et al. 2021

Adv in Health Sci Educ

View full text Add to dashboard Cite

Assessment practices have been increasingly informed by a range of philosophical positions. While generally beneficial, the addition of options can lead to misalignment in the philosophical assumptions associated with different features of assessment (e.g., the nature of constructs and competence, ways of assessing, validation approaches). Such incompatibility can threaten the quality and defensibility of researchers' claims, especially when left implicit. We investigated how authors state and use their philosophical positions when designing and reporting on performance-based assessments (PBA) of intrinsic roles, as well as the (in)compatibility of assumptions across assessment features. Using a representative sample of studies examining PBA of intrinsic roles, we used qualitative content analysis to extract data on how authors enacted their philosophical positions across three key assessment features: (1) construct conceptualizations, (2) assessment activities, and (3) validation methods. We also examined patterns in philosophical positioning across features and studies. In reviewing 32 papers from established peer-reviewed journals, we found (a) authors rarely reported their philosophical positions, meaning underlying assumptions could only be inferred; (b) authors approached features of assessment in variable ways that could be informed by or associated with different philosophical assumptions; (c) we experienced uncertainty in determining (in)compatibility of philosophical assumptions across features. Authors' philosophical positions were often vague or absent in the selected contemporary assessment literature. Leaving such details implicit may lead to misinterpretation by knowledge users wishing to implement, build on, or evaluate the work. As such, assessing claims, quality and defensibility, may increasingly depend more on who is interpreting, rather than what is being interpreted.

show abstract

“…Since the idea of a diagnostic team is relatively new, [29] there is are substantial knowledge gapsand research opportunitieswith respect to how best define and design these teams to optimize the diagnostic process and clinical reasoning outcomes. While there is much known about the assessment of individuals [30] and teams [31], we must develop and gather validity evidence for better tools that allow more meaningful measurement of diagnostic performance. With respect to team assessment, teamwork alone should not be the dependent variable of these assessmentsinstead, teamwork should be treated as an independent variable that leads to a clinically important outcome.…”

Section: The Structure Of Teams In Health Carementioning

confidence: 99%

Teamwork in clinical reasoning – cooperative or parallel play?

et al. 2020

View full text Add to dashboard Cite

Teamwork is fundamental for high-quality clinical reasoning and diagnosis, and many different individuals are involved in the diagnostic process. However, there are substantial gaps in how these individuals work as members of teams and, often, work is done in parallel, rather than in an integrated, collaborative fashion. In order to understand how individuals work together to create knowledge in the clinical context, it is important to consider social cognitive theories, including situated cognition and distributed cognition. In this article, the authors describe existing gaps and then describe these theories as well as common structures of teams in health care and then provide ideas for future study and improvement.

show abstract

Measuring Medical Housestaff Teamwork Performance Using Multiple Direct Observation Instruments: Comparing Apples and Apples

Cited by 10 publications

References 19 publications

The Mindful Manager: Validation of a Rounding Leadership Instrument for Residents

The Mindful Manager: Validation of a Rounding Leadership Instrument for Residents

Implicit and inferred: on the philosophical positions informing assessment science

Teamwork in clinical reasoning – cooperative or parallel play?

Contact Info

Product

Resources

About