Interexaminer agreement on the american board of emergency medicine oral certification examination

There is an established expectation that physicians in training demonstrate competence in all aspects of clinical care prior to entering professional practice. Multiple methods have been used to assess competence in patient care, including direct observation, simulation-based assessments, objective structured clinical examinations (OSCEs), global faculty evaluations, 360-degree evaluations, portfolios, self-reflection, clinical performance metrics, and procedure logs. A thorough assessment of competence in patient care requires a mixture of methods, taking into account each method's costs, benefits, and current level of evidence. At the 2012 Academic Emergency Medicine (AEM) consensus conference on educational research, one breakout group reviewed and discussed the evidence supporting various methods of assessing patient care and defined a research agenda for the continued development of specific assessment methods based on current best practices. In this article, the authors review each method's supporting reliability and validity evidence and make specific recommendations for future educational research.ACADEMIC EMERGENCY MEDICINE 2012; 19:1379-1389 by the Society for Academic Emergency Medicine I n 2001, the Accreditation Council for Graduate Medical Education (ACGME) introduced a timeline for the implementation of training and assessment in six core competencies that form the foundation of clinical competence. Introduced in 1996, the Canadian CanMEDS manager competency correlates to the AC-GME patient care competency, broadly defined as "the active engagement in decision-making in the operation of the healthcare system."1 The patient care competency for emergency medicine (EM) has been defined by a previous Academic Emergency Medicine (AEM) consensus conference, 2 now further elaborated on by the milestones in training, 3 as being able to efficiently gather and synthesize medical and diagnostic information, prioritize tasks, and implement management plans on multiple patients, as well as performing essential invasive procedures competently.There is an explicit expectation that physicians in training demonstrate competence in various aspects of clinical care prior to graduation and professional practice. 4 While this accountability falls squarely on the shoulders of residency training programs, it is mirrored by commensurate expectations of maintenance of competency during ongoing professional practice.The goals of the 2012 AEM consensus conference patient care working group were to describe the current state of evidence for assessment of competence in patient care and define a research agenda for the further development of specific assessment methods based on current best practices. METHODSA search was conducted using MEDLINE 1996-present using the key word search terms "assessment," "patient care," "competency," "competence," "assess*," "emergency," and "education" and limited to humans and English language [boolean search: ((assessment and patient care AND (competency or competence)) OR (assess* a...

Section: Overview Of Assessment Methods Identifiedmentioning

confidence: 99%

Section: Overview Of Assessment Methods Identifiedmentioning

confidence: 99%

“…The oral examination format used by ABEM has demonstrated an interexaminer agreement of 97% on critical actions and 95% on performance ratings. 65 Validity evidence has been demonstrated through comparison to other measures, such as the mini-CEX, 26 improvement with increasing levels of training, 3 Global assessments are subject to recall bias, response bias, and the subjectivity of non-clinical factors such as the halo or millstone effects.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Assessing Patient Care: Summary of the Breakout Group on Assessment of Observable Learner Performance

Takayesu

Kulstad

Wallenstein

et al. 2012

“…Past research suggests that these high-stakes summative assessments are reliable and valid measures of performance. 49,50 Many residency programs conduct local or regional practice oral examinations to prepare their graduates for the ABEM oral examination. One study found that performance on the local practice oral examination moderately correlated with the ITE score and ABEM qualifying examination score, but not with the ABEM oral examination score.…”

Section: Objective Structured Clinical Examination (Osce)mentioning

confidence: 99%

Assessing Medical Knowledge of Emergency Medicine Residents

Goyal

Aldeen

Leone

et al. 2012

The Accreditation Council for Graduate Medical Education (ACGME) requires that emergency medicine (EM) residency graduates are competent in the medical knowledge (MK) core competency. EM educators use a number of tools to measure a resident's progress toward this goal; it is not always clear whether these tools provide a valid assessment. A workshop was convened during the 2012 Academic Emergency Medicine consensus conference "Education Research in Emergency Medicine: Opportunities, Challenges, and Strategies for Success" where assessment for each core competency was discussed in detail. This article provides a description of the validity evidence behind current MK assessment tools used in EM and other specialties. Tools in widespread use are discussed, as well as emerging methods that may form valid assessments in the future. Finally, an agenda for future research is proposed to help address gaps in the current understanding of MK assessment.

“…[1][2][3][4][5] As ABEM implements the new testing platform, it seeks to provide validity evidence for the new format. The determination of valid use is an incremental, iterative, and inferential process.…”

mentioning

confidence: 99%

Initial Validity Analysis of the American Board of Emergency Medicine Enhanced Oral Examination

Kowalenko

Heller

Strauss

et al. 2017

Objectives: The American Board of Emergency Medicine (ABEM) has introduced a new testing format for the oral certification examination (OCE): the enhanced oral or "eOral" format. The purpose of this study was to perform initial validity analyses of the eOral format. The two hypotheses were: 1) the case content in the eOral format was sufficiently similar to clinical practice and 2) the eOral case materials were sufficiently similar to clinical practice. The eOral and traditional formats were compared for these characteristics.Methods: This was a prospective survey study. The survey was administered as a voluntary postexamination activity at the end of the 2015 spring (April 25-27) and fall (October 10-13) ABEM OCEs. The survey is a routine part of the ABEM oral examination experience. For 2015, two additional questions were added to gauge the similarity of the eOral format to clinical practice. Validity was defined by content and substantive elements within Messick's model of construct validity as well as portions of Kane's validity model. Results:Of the 1,746 physicians who took the oral examination, 1,380 physicians (79.0%) completed all or part of the study survey questions. The majority of respondents agreed the patient presentations in the cases were similar (strongly agreed or agreed) to cases seen in clinical practice, in both the traditional cases (95.1%) and the eOral cases (90.1%). Likewise, the majority of respondents answered that the case materials (e.g., laboratory, radiographs) were similar (strongly agreed or agreed) to what they encounter in clinical practice, both in the traditional format (85.8%) and in the eOral cases (93.7%).Conclusions: Most emergency physicians reported that the types of cases tested in the traditional and eOral formats were similar to cases encountered in clinical practice. In addition, most physicians found the case materials to be similar to what is seen in clinical practice. This study provides early validity evidence for the eOral format.