Development of a novel behaviourally anchored instrument for the assessment of surgical trainees

Abstract

Background: The Royal Australasian College of Surgeons (RACS) created its competency framework in 2003 which initially consisted of nine competencies each regarded as equally important for a practising surgeon. The JDocs Framework is aligned to these competencies and provides guidance for junior doctors working towards the Surgical Education and Training program. Methods: A novel assessment instrument was designed around the JDocs framework using 48 behaviourally anchored questions. The study was completed in 2020 across five public hospitals in the ACT and NSW. Participants were invited to complete the self-assessment form online. Results: Thirty-six of 59 (61%) trainees participated in the study, with 67 of 68 (98.5%) supervisors having completed the assessment form. Trainee self-rating scores were lower than that of supervisor ratings across all competencies except communication. The selfrating scores were negatively correlated with the seniority of a trainee's level in all nine competencies. The years of post-graduate experience was positively correlated with seven of the nine competencies. For gender and International Medical Graduate status, correlation was only identified for health advocacy and medical expertise. There was no correlation identified with a trainee's age. Conclusion: This pilot study has provided an opportunity to explore a new assessment instrument for surgical trainees that is aligned to the RACS competency framework using behaviourally anchored questions. Looking ahead, a better understanding of this instrument will potentially be helpful in early identification of underperforming trainees in order to facilitate early intervention, or its use as a selection tool for formal training programs.

A quantitative analysis of the perceived barriers to implementation of multisource feedback in surgical training

“…In our original pilot study of the novel instrument, 60% of trainees and 98% of supervisors responded. 10 This was reflected in the scores provided by both supervisors and trainees (Table 1). The electronic delivery method and formatting of our pilot study was able to overcome this barrier.…”

“…The electronic delivery method and formatting of our pilot study was able to overcome this barrier. 10,22 In the original pilot study of the novel instrument, trainees on average took 14 min 36 s to complete the assessment, while supervisors took 26 min 56 s (84% longer). 10 Given the time commitment needed to complete evaluations, it is important to consider whether protected time should be provided to trainees and supervisors when this instrument is no longer research and formally used as part of an assessment program in surgical units.…”

“…10,22 In the original pilot study of the novel instrument, trainees on average took 14 min 36 s to complete the assessment, while supervisors took 26 min 56 s (84% longer). 10 Given the time commitment needed to complete evaluations, it is important to consider whether protected time should be provided to trainees and supervisors when this instrument is no longer research and formally used as part of an assessment program in surgical units. Future iterations of the assessment could explore whether it is possible to further streamline some questions in each competency.…”

“…Participating trainees nominated two supervisors to provide ratings utilizing an analogous modified version of the instrument to suit the supervisor perspective. 10 The original instrument used in the pilot study is included as supplementary data for reference.…”

“…The authors have previously designed and published a novel instrument that was designed to incorporate the Royal Australian College of Surgeons (RACS) original nine competencies utilizing the JDocs framework. 10 It is generally accepted that clinical assessment instruments are superior when the rubric is behaviourally anchored and referencing specific behaviour patterns, rather than generic descriptors of values and principles. [11][12][13][14] In this pilot study, behaviourally anchored questions were used to maximize relevance and accuracy of the RACS competencies to the daily clinical practice of trainees.…”