Development and Implementation of Situational Judgment Tests as an Evaluation Method for Training Oncology Physicians: Application in the KOKON-KTO Training

Abstract

Situational judgment tests (SJTs) are often used in aptitude testing and present practice-specific challenges. Their implementation into online training programs provides the opportunity to assess learning progress and improve training quality. In this study, text-based SJTs for oncology physicians were developed, validated, and implemented into the KOKON-KTO training which uses a blended learning training format to teach oncology physicians how to consult cancer patients on complementary and integrative medicine (CIM). The SJT was implemented to measure the e-learning results. In the development and validation phase, a total of 15 SJTs (each SJT including 1 best choice answer based on training content and 4 distractors; 9 SJTs for oncologists and 6 SJTs for oncology gynecologists only) were developed by an interprofessional team (n=5) using real-case vignettes and applying an in-depth review process. Best answers were validated by experts (oncologists and oncology gynecologists) with experience in advising cancer patients on CIM. In the implementation and evaluation phase, SJTs were answered by KOKON-KTO training participants (n=19) pre- and post e-learning. Results were analyzed using descriptive measurements, item difficulties, and Cohen’s d for effect size pre- and post-training. The experts (n=12, 49.8% gynecologists) agreed with best choice answers (69.4% for oncology gynecology; 81.5% for oncology) in 12 out of 15 SJTs. Comparing pre- and post-training scores, KOKON-KTO training participants were able to improve knowledge substantially (effect sizes for oncologists d=1.7; oncology gynecologists d= .71). Future studies need to increase the number of experts and SJTs in order to apply further psychometric measurements. As part of the KOKON-KTO study, this project is registered as DRKS00012704 on the “German Clinical Trials Register” (Date of registration: 28.08.2017).