The American Board of Obstetrics and Gynecology's remote certifying examination: successes and challenges

Shivraj, Pooja; Chadha, Rajat; Dynis, Deana; Diepenbrock, Kirk; Hoffman, Barbara L; Ramin, Susan M.; Wendel, George D.

doi:10.1016/j.xagr.2022.100136

Cited by 3 publications

(1 citation statement)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The American Board of Obstetrics and Gynecology also recently published the outcome of their web-based oral certification examinations that were also conducted via Zoom [ 30 ]. Between 2021 and 2022, a total of 1491 specialty and 830 subspecialty candidates undertook three 1-hour oral examinations using 3 pairs of examiners per candidate.…”

Section: Discussionmentioning

confidence: 99%

Feasibility and Acceptability of Web-Based Structured Oral Examinations for Postgraduate Certification: Mixed Methods Preliminary Evaluation

Burch,

McGuire,

Buch

et al. 2024

JMIR Form Res

View full text Add to dashboard Cite

Background The COVID-19 pandemic disrupted postgraduate certification examinations globally. The Colleges of Medicine of South Africa continued hosting certification examinations through the pandemic. This was achieved by effecting a rapid transition from in-person to web-based certification examinations. Objective This formative evaluation explored candidates’ acceptability of web-based structured oral examinations (SOEs) hosted via Zoom (Zoom Communications Inc). We also reported the audiovisual quality and technical challenges encountered while using Zoom and candidates’ overall experience with these examinations conducted during the early part of the COVID-19 pandemic. Additionally, performance in web-based certification examinations was compared with previous in-person certification examinations. Methods This mixed methods, single-arm evaluation anonymously gathered candidates’ perceptions of web-based SOE acceptability, audiovisual quality, and overall experience with Zoom using a web-based survey. Pass rates of web-based and previous in-person certification examinations were compared using chi-square tests, with a Yates correction. A thematic analysis approach was adopted for qualitative data. Results Between June 2020 and June 2021, 3105 candidates registered for certification examinations, 293 (9.4%) withdrew, 2812 (90.6%) wrote, and 2799 (99.9%) passed, and 1525 (54.2%) were invited to a further web-based SOE. Examination participation was 96.2% (n=1467). During the first web-based examination cycle (2020), 542 (87.1%) of 622 web-based SOE candidates completed the web-based survey. They reported web-based SOEs as fair (374/542, 69%) and adequately testing their clinical reasoning and insight (396/542, 73.1%). Few would have preferred real patient encounters (173/542, 31.9%) or in-person oral examinations (152/542, 28%). Most found Zoom acceptable (434/542, 80%) and fair (396/542, 73.1%) for hosting web-based SOEs. SOEs resulted in financial (434/542, 80%) and time (428/542, 79%) savings for candidates. Many (336/542, 62%) supported the ongoing use of web-based certification examinations. Only 169 technical challenges in using Zoom were reported, which included connectivity-related issues, poor audio quality, and poor image quality. The thematic analysis identified 4 themes of positive and negative experiences related to web-based SOE station design and content, examination station environment, examiner-candidate interactions, and personal benefits for candidates. Our qualitative analysis identified 10 improvements for future web-based SOEs. Candidates achieved high pass rates in web-based certification examinations in 2020 (1583/1732, 91.39%) and 2021 (850/1067, 79.66%). These were significantly higher (2020: N=8635; χ21=667; P<.001; 2021: N=7988; χ21=178; P<.001) than the previous in-person certification examination pass rate of 58.23% (4030/6921; 2017-2019). Conclusions Web-based SOEs conducted by the Colleges of Medicine of South Africa during the COVID-19 pandemic were well received by candidates, and few technical difficulties were encountered while using Zoom. Better performance was observed in web-based examinations than in previous in-person certification examinations. These early findings support the ongoing use of this assessment method.

show abstract

Section: Discussionmentioning

confidence: 99%

Feasibility and Acceptability of Web-Based Structured Oral Examinations for Postgraduate Certification: Mixed Methods Preliminary Evaluation

Burch,

McGuire,

Buch

et al. 2024

JMIR Form Res

View full text Add to dashboard Cite

show abstract

A comparison of the American Board of Anesthesiology's in-person and virtual objective structured clinical examinations

Sun,

Deiner,

Harman

et al. 2023

Journal of Clinical Anesthesia

View full text Add to dashboard Cite

Administration of the American Board of Anesthesiology’s virtual APPLIED Examination: successes, challenges, and lessons learned

Keegan,

Harman,

McLoughlin

et al. 2024

BMC Med Educ

View full text Add to dashboard Cite

In response to the COVID-19 pandemic, the American Board of Anesthesiology transitioned from in-person to virtual administration of its APPLIED Examination, assessing more than 3000 candidates for certification purposes remotely in 2021. Four hundred examiners were involved in delivering and scoring Standardized Oral Examinations (SOEs) and Objective Structured Clinical Examinations (OSCEs). More than 80% of candidates started their exams on time and stayed connected throughout the exam without any problems. Only 74 (2.5%) SOE and 45 (1.5%) OSCE candidates required rescheduling due to technical difficulties. Of those who experienced “significant issues”, concerns with OSCE technical stations (interpretation of monitors and interpretation of echocardiograms) were reported most frequently (6% of candidates). In contrast, 23% of examiners “sometimes” lost connectivity during their multiple exam sessions, on a continuum from minor inconvenience to inability to continue. 84% of SOE candidates and 89% of OSCE candidates described “smooth” interactions with examiners and standardized patients/standardized clinicians, respectively. However, only 71% of SOE candidates and 75% of OSCE candidates considered themselves to be able to demonstrate their knowledge and skills without obstacles. When compared with their in-person experiences, approximately 40% of SOE examiners considered virtual evaluation to be more difficult than in-person evaluation and believed the remote format negatively affected their development as an examiner. The virtual format was considered to be less secure by 56% and 40% of SOE and OSCE examiners, respectively. The retirement of exam materials used virtually due to concern for compromise had implications for subsequent exam development. The return to in-person exams in 2022 was prompted by multiple factors, especially concerns regarding standardization and security. The technology is not yet perfect, especially for testing in-person communication skills and displaying dynamic exam materials. Nevertheless, the American Board of Anesthesiology’s experience demonstrated the feasibility of conducting large-scale, high-stakes oral and performance exams in a virtual format and highlighted the adaptability and dedication of candidates, examiners, and administering board staff.

show abstract

The American Board of Obstetrics and Gynecology's remote certifying examination: successes and challenges

Cited by 3 publications

References 3 publications

Feasibility and Acceptability of Web-Based Structured Oral Examinations for Postgraduate Certification: Mixed Methods Preliminary Evaluation

Feasibility and Acceptability of Web-Based Structured Oral Examinations for Postgraduate Certification: Mixed Methods Preliminary Evaluation

A comparison of the American Board of Anesthesiology's in-person and virtual objective structured clinical examinations

Administration of the American Board of Anesthesiology’s virtual APPLIED Examination: successes, challenges, and lessons learned

Contact Info

Product

Resources

About