Validation of Multisource Feedback in Assessing Medical Performance: A Systematic Review

Stevens, Sebastian; Read, James H.; Baines, Rebecca; Chatterjee, Arunangsu; Archer, Julian

doi:10.1097/ceh.0000000000000219

Cited by 21 publications

(27 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A further strength of the study is that we described not only the internal structure but also the response process and consequences in detail, thus addressing a gap in the literature. A recent review on the validity of MSF summarized that these aspects were reported too seldom, and as a result, knowledge on these was still limited [51].…”

Section: Discussionmentioning

confidence: 99%

A german-language competency-based multisource feedback instrument for residents: development and validity evidence

et al. 2020

View full text Add to dashboard Cite

Background In medical settings, multisource feedback (MSF) is a recognised method of formative assessment. It collects feedback on a doctor’s performance from several perspectives in the form of questionnaires. Yet, no validated MSF questionnaire has been publicly available in German. Thus, we aimed to develop a German MSF questionnaire based on the CanMEDS roles and to investigate the evidence of its validity. Methods We developed a competency-based MSF questionnaire in German, informed by the literature and expert input. Four sources of validity evidence were investigated: (i) Content was examined based on MSF literature, blueprints of competency, and expert-team discussions. (ii) The response process was supported by analysis of a think-aloud study, narrative comments, “unable to comment” ratings and evaluation data. (iii) The internal structure was assessed by exploratory factor analysis, and inter-rater reliability by generalisability analysis. Data were collected during two runs of MSF, in which 47 residents were evaluated once (first run) or several times (second and third run) on 81 occasions of MSF. (iv) To investigate consequences, we analysed the residents’ learning goals and the progress as reported via MSF. Results Our resulting MSF questionnaire (MSF-RG) consists of 15 items and one global rating, which are each rated on a scale and accompanied by a field for narrative comments and cover a construct of a physician’s competence. Additionally, there are five open questions for further suggestions. Investigation of validity evidence revealed that: (i) The expert group agreed that the content comprehensively addresses clinical competence; (ii) The response processes indicated that the questions are understood as intended and supported the acceptance and usability; (iii) For the second run, factor analysis showed a one-factor solution, a Cronbach’s alpha of 0.951 and an inter-rater reliability of 0.797 with 12 raters; (iv) There are indications that residents benefitted, considering their individual learning goals and based on their ratings reported via MSF itself. Conclusions To support residency training with multisource feedback, we developed a German MSF questionnaire (MSF-RG), which is supported by four sources of validity evidence. This MSF questionnaire may be useful to implement MSF in residency training in German-speaking regions.

show abstract

Section: Discussionmentioning

confidence: 99%

A german-language competency-based multisource feedback instrument for residents: development and validity evidence

et al. 2020

View full text Add to dashboard Cite

show abstract

“…It is important as well that feedback collected by means of MSF is given to the assessed doctor in a facilitated fashion, either by an appraiser, mentor, facilitator, or coach. This, as well as the quality of this facilitation, has been suggested to have a role in the physicians' acceptance and subsequent behavioral change by a review of effectiveness of multi-source feedback [45].…”

Section: Patient Questionnairementioning

confidence: 99%

Assessing anaesthesiology and intensive care specialty physicians: An Italian language multisource feedback system

et al. 2021

View full text Add to dashboard Cite

Background Physician professionalism, including anaesthesiologists and intensive care doctors, should be continuously assessed during training and subsequent clinical practice. Multi-source feedback (MSF) is an assessment system in which healthcare professionals are assessed on several constructs (e.g., communication, professionalism, etc.) by multiple people (medical colleagues, coworkers, patients, self) in their sphere of influence. MSF has gained widespread acceptance for both formative and summative assessment of professionalism for reflecting on how to improve clinical practice. Methods Instrument development and psychometric analysis (feasibility, reliability, construct validity via exploratory factor analysis) for MSF questionnaires in a postgraduate specialty training in Anaesthesiology and intensive care in Italy. Sixty-four residents at the Università del Piemonte Orientale (Italy) Anesthesiology Residency Program. Main outcomes assessed were: development and psychometric testing of 4 questionnaires: self, medical colleague, coworker and patient assessment. Results Overall 605 medical colleague questionnaires (mean of 9.3 ±1.9) and 543 coworker surveys (mean 8.4 ±1.4) were collected providing high mean ratings for all items (> 4.0 /5.0). The self-assessment item mean score ranged from 3.1 to 4.3. Patient questionnaires (n = 308) were returned from 31 residents (40%; mean 9.9 ± 6.2). Three items had high percentages of “unable to assess” (> 15%) in coworker questionnaires. Factor analyses resulted in a two-factor solution: clinical management with leadership and accountability accounting for at least 75% of the total variance for the medical colleague and coworker’s survey with high internal consistency reliability (Cronbach’s α > 0.9). Patient’s questionnaires had a low return rate, a limited exploratory analysis was performed. Conclusions We provide a feasible and reliable Italian language MSF instrument with evidence of construct validity for the self, coworkers and medical colleague. Patient feedback was difficult to collect in our setting.

show abstract

“…9 10 12 19 One method of assessment called 360° feedback or multisource feedback (MSF), requires evaluations that are completed by multiple people in a person's sphere of influence. 20 Literature examining MSF assessment in Canadian residents has left the CanMEDS role of Leader underexplored. Around the world, efforts to develop an MSF tool to assess leadership have not specifically assessed the Leader role, have failed to seek feedback from patients or have done neither.…”

Section: Original Researchmentioning

confidence: 99%

“…Around the world, efforts to develop an MSF tool to assess leadership have not specifically assessed the Leader role, have failed to seek feedback from patients or have done neither. [20][21][22][23][24] Our research question was, 'What does the construct of leadership look like in junior residents using an MSF tool in a healthcare setting?' The purpose of this study was to describe the construct of leadership in junior resident physicians by implementing a new MSF survey tool called the Learning by Evaluation from All-inclusive 360 Degree Engagement of Residents (LEADER) tool.…”

Section: Original Researchmentioning

confidence: 99%

Assessing leadership in junior resident physicians: using a new multisource feedback tool to measure Learning by Evaluation from All-inclusive 360 Degree Engagement of Residents (LEADER)

Bharwani

Swystun

Paolucci

et al. 2020

leader

View full text Add to dashboard Cite

BackgroundThe multifaceted nature of leadership as a construct has implications for measuring leadership as a competency in junior residents in healthcare settings. In Canada, the Royal College of Physicians and Surgeons of Canada’s CanMEDS physician competency framework includes the Leader role calling for resident physicians to demonstrate collaborative leadership and management within the healthcare system. The purpose of this study was to explore the construct of leadership in junior resident physicians using a new multisource feedback tool.MethodsTo develop and test the Learning by Evaluation from All-Inclusive 360 Degree Engagement of Residents (LEADER) Questionnaire, we used both qualitative and quantitative research methods in a multiphase study. Multiple assessors including peer residents, attending physicians, nurses, patients/family members and allied healthcare providers as well as residents’ own self-assessments were gathered in healthcare settings across three residency programmes: internal medicine, general surgery and paediatrics. Data from the LEADER were analysed then triangulated using a convergent-parallel mixed-methods study design.ResultsThere were 230 assessments completed for 27 residents. Based on key concepts of the Leader role, two subscales emerged: (1) Personal leadership skills subscale (Cronbach’s alpha=0.81) and (2) Physicians as active participant-architects within the healthcare system (abbreviated to active participant-architects subscale, Cronbach’s alpha=0.78). There were seven main themes elicited from the qualitative data which were analogous to the five remaining intrinsic CanMEDS roles. The remaining two themes were related to (1) personal attributes unique to the junior resident and (2) skills related to management and administration.ConclusionsFor healthcare organisations that aspire to be proactive rather than reactive, we make three recommendations to develop leadership competence in junior physicians: (1) teach and assess leadership early in training, (2) empower patients to lead and transform training and care by evaluating doctors, (3) activate frontline care providers to be leaders by embracing patient and team feedback.

show abstract

Validation of Multisource Feedback in Assessing Medical Performance: A Systematic Review

Cited by 21 publications

References 29 publications

A german-language competency-based multisource feedback instrument for residents: development and validity evidence

A german-language competency-based multisource feedback instrument for residents: development and validity evidence

Assessing anaesthesiology and intensive care specialty physicians: An Italian language multisource feedback system

Assessing leadership in junior resident physicians: using a new multisource feedback tool to measure Learning by Evaluation from All-inclusive 360 Degree Engagement of Residents (LEADER)

Contact Info

Product

Resources

About