Applying Inter-rater Reliability to Improve Consistency in Classifying PhD Career Outcomes

Abstract

In the past year, there has been an exciting groundswell of national efforts to integrate multiple taxonomies for the transparent dissemination and analysis of PhD career outcomes. In this study, we leveraged the unique resources of the Broadening Experiences in Scientific Training Consortium to examine the reliability of the three-tiered Unified Career Outcomes Taxonomy (UCOT v.2017) that was collaboratively developed at a meeting convened by Rescuing Biomedical Research in August 2017. Using an amended version of the UCOT v.2017 (UCOT v.2017-rev1) and a new Supplementary Guidance document, we categorized over 570 PhD alumni records from three different universities. Utilizing Krippendorff's alpha to measure the interrater reliability from nine different individuals, we determined moderate to robust reproducibility within the first two tiers of the taxonomy (Workforce Sector and Career Type); however, the reliability for the third tier (Job Function) did not meet established standards. The team identified significant sources of error, revised category definitions, improved coder training materials and processes, and tested for improved reliability through coding 219 PhD alumni records using the revised taxonomy, UCOT v.2017-rev2. Our results revealed that the changes introduced in UCOT v.2017-rev2 improved inter-rater reliability in all three tiers, and either met or exceeded the acceptable standards for reliability. A final set of clarifications were made to UCOT v.2017-rev2, resulting in UCOT v.2018 and a Finalized Guidance document. Our findings underscore the importance of carefully developing guidance documents to aid coders in the reliable and consistent categorization of alumni career outcomes. We propose periodic assessment of the UCOT v.2018 to address the natural evolution of PhD careers in the global workforce. Ultimately, we hope that UCOT v.2018 will aid in the classification and dissemination of alumni career outcomes that is essential to educating trainees, institutions, and agencies about the diversity of career options for PhDs, and therein empower all PhDs to pursue the careers of their choice.

The NIH “BEST” programs: Institutional programs, the program evaluation, and early data

“…One collaboration among a group of BEST awardees resulted in development of a career taxonomy that was adapted and adopted by a broader group of organizations as a unified taxonomy for biomedical careers and is now used by a number of academic institutions and the NGLS. A more recent follow‐up study from this collaboration of BEST sites tested the reliability of the taxonomy tool and present it in a user friendly format …”

“…A more recent follow-up study from this collaboration of BEST sites tested the reliability of the taxonomy tool and present it in a user friendly format. 28 An ambitious goal of the BEST awards was to sustain successful activities after the 5-year non-renewable awards end. All sites have accomplished this by either securing funding for BEST activities and associated staff or integrating the activities into their graduate programs.…”