A review of multi-source feedback focusing on psychometrics, pitfalls and some possible solutions

Abstract

With the re-emergence of competency-based frameworks in professional education, multisource feedback (MSF) has become a common method for assessing various competencies, including communication, professionalism, and aspects of teambased performance. A wide variety of publications over the past 50 years or more in the business and health literature would seem to support the use of MSF at least for quality improvement (QI) purposes. However, our own experience with using MSF in physicians has been quite mixed, with some physicians embracing the experience and making improvements and others the complete opposite. We decided to review the existing literature on MSF to try to identify key aspects of successful MSF programs. This paper presents a structured critique of the literature on the use of MSF in physician populations. The findings were surprising as key assumptions around the validity and reliability of MSF were not consistently met, key lessons from earlier research were not carried over to present day programs and perhaps most concerning was a lack of evidence for MSF producing meaningful sustained behavior change. From these findings we suggest some key areas of potential improvement in MSF programs.

Multisource feedback: an overview of its use and application as a formative assessment

“…Facilitated discussions are critical to guide the physician or learner in the co-development of the learning change plan and its follow-up. [11][12][13] These studies and other publications 1,2,30,43,48,49 have demonstrated the importance of discussion and follow-up and are informing best practices for MSF. The de-briefer or facilitator requires training and may be someone appointed for this purpose for medical students and residents so that they are at armslength from other evaluation and assessment processes or it may be a preceptor or supervisor.…”

Künstliche Intelligenz bei lamellierenden Keratoplastiken

“…The study of Yousefi et al [17] addressed the question of whether an AI algorithm trained using 12 242 OCT scans of the corneas of 3162 patients with corneal conditions and 333 eyes before keratoplasty (penetrating keratoplasty, DALK, DMEK, DSAEK) can predict the probability of the need for keratoplasty. A number of different clusters were created, each of which was capable of predicting the normalized probability for the respective surgical procedure for the cohort under investigation [4].…”

“…Eine Studie von Yousefi et al 17 untersuchte generell, ob ein anhand von 12 242 OCT-Scans der Hornhaut von 3162 Patienten mit einer Hornhauterkrankung und 333 Augen vor einer Keratoplastik (penetrierende Keratoplastik, DALK, DMEK, DSAEK) trainierter KI-Algorithmus die Wahrscheinlichkeit der Notwendigkeit einer Keratoplastik vorhersagen kann. Dabei konnten unterschiedliche Cluster gebildet werden, die für die untersuchte Kohorte jeweils die normalisierte Wahrscheinlichkeit für das jeweilige OP-Verfahren vorhersagen konnte 4 .…”

Statistical indicators of compliance with anti-COVID-19 public health measures at European airports

“…Although the information source was self-reported in all cases, it was considered a multisource because it came from three different populations. Some researchers also use the term multilevel; however, in this study, each population cannot be considered a “level” because they do not form independent clusters [ 27 ]. In this way, if only passengers' statements were considered, the measurement would not be complete or exhaustive.…”