Applying inter-rater reliability to improve consistency in classifying PhD career outcomes

Abstract

Background: There has been a groundswell of national support for transparent tracking and dissemination of PhD career outcomes. In 2017, individuals from multiple institutions and professional organizations met to create the Unified Career Outcomes Taxonomy (UCOT 2017), a three-tiered taxonomy to help institutions uniformly classify career outcomes of PhD graduates. Early adopters of UCOT 2017, noted ambiguity in some categories of the career taxonomy, raising questions about its consistent application within and across institutions. Methods: To test and evaluate the consistency of UCOT 2017, we calculated inter-rater reliability across two rounds of iterative refinement of the career taxonomy, classifying over 800 PhD alumni records via nine coders. Results: We identified areas of discordance in the taxonomy, and progressively refined UCOT 2017 and an accompanying Guidance Document to improve inter-rater reliability across all three tiers of the career taxonomy. However, differing interpretations of the classifications, especially for faculty classifications in the third tier, resulted in continued discordance among the coders. We addressed this discordance with clarifying language in the Guidance Document, and proposed the addition of a flag system for identification of the title, rank, and prefix of faculty members. This labeling system provides the additional benefit of highlighting the granularity and the intersectionality of faculty job functions, while maintaining the ability to sort by - and report data on - faculty and postdoctoral trainee roles, as is required by some national and federal reporting guidelines. We provide specific crosswalk guidance for how a user may choose to incorporate our suggestions while maintaining the ability to report in accordance with UCOT 2017. Conclusions: Our findings underscore the importance of detailed guidance documents, coder training, and periodic collaborative review of career outcomes taxonomies as PhD careers evolve in the global workforce. Implications for coder-training and use of novice coders are also discussed.

From goal to outcome: Analyzing the progression of biomedical sciences PhD careers in a longitudinal study using an expanded taxonomy

“…The classification system we developed is similar to the three‐tiered Unified Career Outcomes Taxonomy 9 and further optimized by Stayart et al (2020), 10 with differences described in Appendix S1. At least two experienced career outcomes professionals independently categorized every job, and every position that was not identically classified was reviewed to reach a consensus category.…”

Making Strides in Doctoral-Level Career Outcomes Reporting: Surveying the Landscape of Classification and Visualization Methodologies and Creating a Crosswalk Tool

“…The UCOT provided an initial set of standardized definitions to common terms, which were later empirically tested and clarified to address identified areas of uncertainty. The taxonomy was iteratively tested in Stayart et al 2020 13 to determine the classification consistency across different ‘raters’; this work resulted in a supplemental guidance document on how to interpret various cases, such that definitions would be applied consistently by practitioners who were curating the data. The results of Stayart et al (2020) 13 suggested that reliability improved with all tiers, and improvement occurred even when using non-experienced coders; this experimentally tested, updated version of the UCOT taxonomy was termed UCOT-Exp2.…”

“…It can be adapted to track those in other disciplines beyond STEM. As an example, Wayne State University has adapted UCOT to the humanities by replacing ‘science-related’ with ‘discipline-related.’ The third tier of the taxonomy can be further adapted by adding additional job functions that are applicable to disciplines outside of STEM 13,14 .…”

“…The U of T has also joined the Coalition for Next Generation Life Science (CNGLS), and they have updated their interactive dashboard to report their career outcomes according to the standards set for those joining CNGLS, which includes reporting via the UCOT 2017 taxonomy format. Reporting data based on these two different taxonomies allows readers to see how similar the Canadian employment sectors are relative to those within the 2017 UCOT (http://rescuingbiomedicalresearch.org/rbr-actions/improving-transparency-ph-d-career-outcomes/), and further updated (UCOT Exp2) based on experimental evidence in Stayart et al 2020 13 .…”

“…One suggestion for how to clearly delineate this would be to develop a universal shorthand methodology for tagging all reports and graphics with the taxonomy used. For example, consider the UCOT taxonomy-when the taxonomy was revised based on feedback and tests to ensure inter-rater reliability, a new version was published 13 . The original version is referred to as UCOT 2017, and the updated version UCOT-Experimental (UCOT-Exp2).…”