Applying inter-rater reliability to improve consistency in classifying PhD career outcomes

Abstract

Background: There has been a groundswell of national support for transparent tracking and dissemination of PhD career outcomes. In 2017, individuals from multiple institutions and professional organizations met to create the Unified Career Outcomes Taxonomy (UCOT 2017), a three-tiered taxonomy to help institutions uniformly classify career outcomes of PhD graduates. Early adopters of UCOT 2017, noted ambiguity in some categories of the career taxonomy, raising questions about its consistent application within and across institutions. Methods: To test and evaluate the consistency of UCOT 2017, we calculated inter-rater reliability across two rounds of iterative refinement of the career taxonomy, classifying over 800 PhD alumni records via nine coders. Results: We identified areas of discordance in the taxonomy, and progressively refined UCOT 2017 and an accompanying Guidance Document to improve inter-rater reliability across all three tiers of the career taxonomy. However, differing interpretations of the classifications, especially for faculty classifications in the third tier, resulted in continued discordance among the coders. We addressed this discordance with clarifying language in the Guidance Document, and proposed the addition of a flag system for identification of the title, rank, and prefix of faculty members. This labeling system provides the additional benefit of highlighting the granularity and the intersectionality of faculty job functions, while maintaining the ability to sort by - and report data on - faculty and postdoctoral trainee roles, as is required by some national and federal reporting guidelines. We provide specific crosswalk guidance for how a user may choose to incorporate our suggestions while maintaining the ability to report in accordance with UCOT 2017. Conclusions: Our findings underscore the importance of detailed guidance documents, coder training, and periodic collaborative review of career outcomes taxonomies as PhD careers evolve in the global workforce. Implications for coder-training and use of novice coders are also discussed.

Career planning courses increase career readiness of graduate and postdoctoral trainees

“…In addition, while the data is inconclusive, administrative career-focused staff (e.g., career and professional development staff) may have become a newly utilized resource – and almost certainly added to the potential mentor-pool from which trainees may choose to draw. The development of additional resources to support doctoral trainee career transitions (e.g., NIH BEST programs; Lara et al, 2020 ; Lenzi et al, 2020 ; NIH, 2012 ) is of growing national interest (e.g., [ Blank et al, 2017 ; Benderly, 2015 ; FOBGAPT, 2017 ; Stayart et al, 2018 ); one solution to workforce preparation in graduate education includes the use of staff trained to provide guidance and resources leading up to and during this transition to the workforce. Furthermore, the focus on career transitions and career outcomes is especially salient currently due to the federal focus that has recently been introduced to encourage development of more such programs (e.g., federal training grants such as NIH/NIGMS T-32 including career development training requirements and career outcome reporting; Gammie et al, 2017 ); hence, it seems likely that the importance of shepherding trainees through career transitions will only continue to gain importance.…”

Creating and sustaining collaborative multi-institutional industry site visit programs: a toolkit

“…Career outcome data were then grouped into taxonomic tiers jointly agreed upon by the authors, particularly those corresponding to Tier 1 of the commonly used PhD Career Outcomes Taxonomies ( Lenzi et al, 2020 ; Stayart et al, 2020 ; Xu et al, 2018 ). Furthermore, due to the specific types of companies that hosted site visits, we further categorized each company into either Contract Service Organization (CSO), Biotech, or Pharma to reflect the Research Triangle Park’s primary markets.…”

“…It is widely known that the number of tenure-track positions remains relatively flat while the number of PhD-holders increases ( National Institutes of Health [NIH], 2012 )—meaning many individuals will enter into other types of careers beyond faculty positions ( Stayart et al, 2020 ). Therefore, institutions should be preparing their graduate students and postdoctoral scholars for these other types of careers ( Sinche et al, 2017 )—and this idea finally seems to be gaining traction in academe ( Lenzi et al, 2020 ).…”

A crosswalk to align the GSS and NIH grants databases based on university names

“…For this reason and others, multiple reports on the research enterprise, and the biomedical research enterprise in particular, recommended the federal government improve data collection on the research workforce (5). Recently, universities began collecting and publishing the career outcomes of their Ph.D. and postdoc alumni (6)(7)(8)(9). These efforts satisfy some of the recommendations from prior reports calling for more extensive data collection on the research workforce, but they do not solve the entire problem.…”