A multisource feedback tool to assess ward round leadership skills of senior paediatric trainees: (2) Testing reliability and practicability

Abstract

Background A five-domain multisource feedback (MSF) tool was previously developed in 2009-2010 by the authors to assess senior paediatric trainees' ward round leadership skills. Objectives To determine whether this MSF tool is practicable and reliable, whether individuals' feedback varies over time and trainees' views of the tool. Methods The MSF tool was piloted (April-July 2011) and field tested (September 2011-February 2013) with senior paediatric trainees. A focus group held at the end of field testing obtained trainees' views of the tool. Results In field testing, 96/115 (84%) trainees returned 633 individual assessments from three different ward rounds over 18 months. The MSF tool had high reliability (Cronbach's α 0.84, G coefficient 0.8 for three raters). In all five domains, data were shifted to the right with scores of 3 (good) and 4 (excellent). Consultants gave significantly lower scores ( p<0.001), as did trainees for self-assessment ( p<0.001). There was no significant change in MSF scores over 18 months but comments showed that trainees' performance improved. Trainees valued these comments and the MSF tool but had concerns about time taken for feedback and confusion about tool use and the paediatric assessment strategy. Conclusions A five-domain MSF tool was found to be reliable on pilot and field testing, practicable to use and liked by trainees. Comments on performance were more helpful than scores in giving trainees feedback.

Discipline-specific competency-based curricula for leadership learning in medical specialty training

“…The medical disciplines represented in the 32 papers were: anaesthesia (Edler, Adamshick, Fanning and Piro, 2010;Lattore and Lumb, 2005), emergency medicine (Thoma et al, 2015;Uddin, Barnett, Parker, Links and Alexander, 2008) dermatology (Baird, Soldanska, and Anderson, 2012), general practice (Hadly, 2010; Gillam and Khanchandani, 2013;Khanchandani and Gillam, 2012;Patterson, Godden, Rughani and Smithson, 2013), internal medicine (Angus, 2009;Hunziker, Tschan, Semmer and Marsch, 2013;Moore, Wininger, and Martin, 2016), obstetrics and gynaecology (Scheele et al, 2014;Steinhardt, 2015), pathology (Brimhall et al, 2007;Hemmer et al, 2007), psychiatry (Garg et al, 2011;Malloy, Butt and Sorter, 2010;Stergiopolous, Maggi, and Sockalingam, 2009), paediatrics (Goodyear, Lakshminarayana, Wall, and Bindal, 2015;Hicks, P., Schumacher, Guralnick, Carraccio and Burke, 2014;Kuo, Thyne, Chen, West and Kamei, 2010;Lakshminarayana, Wall, Bindal and Goodyear, 2015), public health (Foster et al, 2008;Smith, 2015), maternal and child health (Department of Health and Human Services, 2009;Humphreys et al, 2015), medical administration (MacCarrick, Owen, and Hearder, 2014), radiation oncology (Turner et al, 2017), radiology (Brandon and Mullen, 2013;Donelly, 2015;Lexa, 2011), and surgery (Awad, Hayley, Fagan, Berger, and Brunicardi, 2004;Jaffe, Pradarelli, Lemak, Mulholland and Dimick, 2016;…”

A qualitative exploration: questioning multisource feedback in residency education

“…Further explorations into the effects of feedback content and reactions to it [ 3 , 10 , 11 ], coaching [ 12 , 13 ], physician characteristics [ 1 ], and other variables that influence performance improvement [ 14 ], have had little practical application. Recent studies have begun to explore the relationships between residents and allied healthcare professionals (AHCPs) during inpatient ward rounds [ 15 ]. Although evidence from both the feedback and MSF literature emphasizes the source of feedback as a major factor, there has been minimal direct investigation into the nature of the relationship between feedback provider and recipient and how it may affect MSF [ 16 , 17 ].…”

Assessing leadership in junior resident physicians: using a new multisource feedback tool to measure Learning by Evaluation from All-inclusive 360 Degree Engagement of Residents (LEADER)

“…First, a generalisability theory study could not be used because we were not able to control for the number of assessor-occasion combinations, which impaired our confidence to extrapolate the results of this study. 22 24 Second, we were unable to establish divergence and congruence with other measures to contribute additional validity evidence for the use of the LEADER tool. Third, we do not know if the feedback provided to the residents led to any changes in their practice (consequential validity) 26 since we do not have data on impact and application of the tool.…”

“…Around the world, efforts to develop an MSF tool to assess leadership have not specifically assessed the Leader role, have failed to seek feedback from patients or have done neither. [20][21][22][23][24] Our research question was, 'What does the construct of leadership look like in junior residents using an MSF tool in a healthcare setting?' The purpose of this study was to describe the construct of leadership in junior resident physicians by implementing a new MSF survey tool called the Learning by Evaluation from All-inclusive 360 Degree Engagement of Residents (LEADER) tool.…”