Direct Observation in Medical Education: A Review of the Literature and Evidence for Validity

Abstract

In 2000, the Accreditation Council for Medical Education introduced a new initiative that substantively changed the method by which residency programs are evaluated. In this new competency-based approach to residency education, assessment of performance became a main area of interest, and direct observation was offered as a tool to assess knowledge and skills. Despite being an inherent part of medical education as faculty and learners work together in clinical experiences, direct observation has traditionally been an informal and underused assessment method across all specialties. Residents and students report rarely being observed during their educational process, even though they value the experience. Reasons for this include a lack of faculty time, a lack of faculty skills, a potential stressful effect on the learner, and a perceived lack of validation of the assessment. This article examines the literature regarding the use of direct observation in medical education with a focus on validity evidence. We performed a PubMed search of articles pertaining to direct observation, using key words such as direct observation, performance observation, clinical observation, students, and residents. A subsequent search was conducted in known articles, focusing on variations of the term observation in the titles of articles and introducing the concept of clinical competence. In conclusion, direct observation is a unique and useful tool in the assessment of medical students and residents. Assessing learners in natural settings offers the opportunity to see beyond what they know and into what they actually do, which is fundamentally essential to training qualified physicians. Although the literature identifies several threats to its validity as an assessment, it also demonstrates methods to minimize those threats. Based on the current recommendations and need for performance assessment in education and with attention paid to the development and design, direct observation can and should be included in medical education curricula.

Teaching at the Bedside. Maximal Impact in Minimal Time

“…The final part of bedside teaching includes closing the loop with the learners by using direct observations to generate useful feedback (19). This practice has become an essential part of undergraduate and graduate medical education given the recent emphasis on assessing milestones and performance on entrustable professional activities (20)(21)(22).…”

Methods of assessment used by osteopathic educational institutions

“…The concept of high face validity with this type of assessment is widely supported in the literature, [24][25][26][27][28] however both content validity 7 and reliability are questioned by some authors. This is particularly in relation to the heterogeneity of cases, 14,21,27 limited sampling of patients, 21 and lack of standardisation.…”

“…35 However, the institutions were also mindful of the fact that just because a practitioner was qualified and experienced, this did not automatically make them an appropriate examiner. 36 Previous research suggests that inadequately prepared examiners impact on the reliability of an assessment, 24 and that examiner training improves both the reliability and validity of an examination. 27,36 Examiner training was mostly addressed using an apprenticeship approach, whereby new examiners would observe the process initially, then undertake assessments after a period of time.…”

Evaluación de las competencias clínicas en una residencia de pediatría con el Mini-CEX (Mini-Clinical Evaluation Exercise)

“…15 La observación directa es uno de los instrumentos más útiles para evaluar el desarrollo de las habilidades clínicas de los residentes y el rol de los educadores es fundamental, por lo que la formación de los docentes y la provisión de devolución son críticas para el éxito de un programa de observación directa. 25,26 Una preocupación constante de los educadores médicos debe ser el impacto educativo de los mé-todos de evaluación y de allí la importancia que reviste la devolución. Ésta mejora el aprendizaje, favorece la motivación y las habilidades de autorregulación, es decir la capacidad para reconocer debilidades y fortalezas y así identificar áreas que requieran mejor desempeño.…”