Evaluation of a new e‐learning resource for calibrating OSCE examiners on the use of rating scales

Abstract

Introduction: Rating scales have been described as better at assessing behaviours such as professionalism during Objective Structured Clinical Examinations (OSCEs).However, there is an increased need to train and calibrate staff on their use prior to student assessment. Material and methods:An online e-learning package was developed and made available to all examiners at the Institute of Dentistry at the University of Aberdeen. The package included videos of three OSCE stations (medical emergency, rubber dam placement and handling a complaint) which were recorded in two different scenarios; (excellent and unsatisfactory candidate). These videos were recorded to meet a pre-defined marking score. The examiners were required to mark the six videos using pre-set marking criteria (checklist and rating scales). The rating scales included professionalism, general clinical ability and/or communication skills. For each video, examiners were given four possible options (unsatisfactory, borderline, satisfactory or excellent), and they were provided with a description for each domain. They were also required to complete a questionnaire to gather their views on the use of this elearning environment.Results: Fifteen examiners completed the task. The total scores given were very similar to the expected scores for the medical emergency and complaint stations; however, this was not the case for the rubber dam station (P-value .017 and .036).This could be attributed to some aspects of the placement of the rubber dam being unclear as commented on in the examiners questionnaires. There was consistency in the selection of marks on the rating scales (inter-examiner correlation ranged between 0.916 and 0.979). Conclusion:Further studies are required on the field of e-learning training to calibrate examiners for practical assessment; however, this study provides preliminary evidence to support the use of videos as part of an online training package to calibrate OSCE examiners on the use of rating scales. K E Y W O R D S examiner training, objective structured clinical examination, rating scales | 277 MORENO-LÓPEZ aNd SINCLaIR

Effectiveness of enhanced video‐based learning on removable partial denture module

“…[5][6][7] Learning through videos for knowledge and clinical skills in dentistry saves time and costs because it reduces the amount of a tutor's workload 8 and demand for faculty resources. 9 It can be used as a calibration instrument 10 to improve students' comprehension, 11,12 assist in mental preparation 13 and influence their practical performance. 14 Some studies have found that practical performance is not improved, or knowledge levels between video and conventional learning have no difference 15,16 Numerous studies have reported that students prefer a live clinical demonstration.…”

Students at Saarland University dental school—A survey on their background and curriculum perception

“…Whilst objectively structured theoretical and practical examinations 18,19 are well established in preclinical courses, leading to a high level of comparability amongst students, patient‐based examinations may be seen controversial. It is impossible to organise identical patients for conducting a practical student examination requiring patient treatment.…”

“…Due to the interdependent relationships between students, patients, instructors and administrators, 12 novel curricula should ensure the creation of an educational environment which is satisfying to future dentists 13,14 . Besides lecture formats which have been shown to be favourably organised in the form of problem based learning, 15 assessment of students' performance is also critical in this context with varying results being reported for traditional test methods 16,17 and objectively structured clinical examinations (OSCE) 18,19 …”

Dental students’ perceptions and educational impact of preclinical interactive videos compared and in combination with live demonstrations

“…They can offer students a useful means of learning and a flexible resource to review content at their own pace and according to their individual learning needs 6–9 . Furthermore, videos have been utilized to standardize the delivery of information across tutors 10 and calibrate examiners on the use of rating scales 11 . Videos can also help ensure the consistency of messaging and content when multiple learning sessions are required in large classes 3 …”

“…[6][7][8][9] Furthermore, videos have been utilized to standardize the delivery of information across tutors 10 and calibrate examiners on the use of rating scales. 11 Videos can also help ensure the consistency of messaging and content when multiple learning sessions are required in large classes. 3 However, the use of multimedia learning presents both opportunities and challenges.…”