Pathways for assessing interdisciplinarity: A systematic review

Abstract

In many sectors and in many respects, interdisciplinarity has made progress in recent decades, but less so when it comes to evaluating interdisciplinary work. Challenges remain, especially regarding what counts as ‘good’ interdisciplinarity. While previous reviews describe broad trends contributing to these challenges, high-level observations are less suited to: (1) pinpointing underlying sources of persistent evaluative issues, (2) capturing less frequent counterexamples and emerging trends, and (3) providing practical guidance for moving the field forward. This article presents a novel approach for excavating and analyzing assessment designs within and across a wide range of published studies. It is the first study to structure its review of interdisciplinary evaluation around the basics of evaluation theory, creating a simplified model of assessment design choices called an assessment pathway. It further stands alone as an extensive systematic review of the interdisciplinary literature uniquely placed within the context of evaluation, encompassing 20 years, 142 studies, 1,006 assessment pathways, and 19,114 data points. Results show that while interdisciplinary assessments to date excel at variation, only 12% of identified pathways contain the minimum elements required to reach an evaluative judgment. Increased attention to evaluation theory can catalyze progress and provide structure and shared language to a highly varied domain. Readers can visually explore the full dataset online and download it from Harvard’s Dataverse.

Novel approach for tracking interdisciplinary research productivity using institutional databases

“…Overall, it is more difficult, expensive, and time consuming to work in research teams vs. solo research [5]. At the same time, there is growing evidence that IDRTs are associated with significant advantages such as greater practical impact, visibility, innovation, and potential to secure funding [8]. This evidence suggests that interdisciplinary teamwork should not be viewed as a “one size fits all” magic solution to academic productivity but rather as a strategic approach that should be used thoughtfully for research questions and organizations that are more likely to benefit from IDRT.…”

“…Team Science Cores also develop and implement numerous innovative strategies to train researchers how to be effective leaders and contributors in research teams including approaches aimed at increasing diversity and inclusion in clinical and translational research. However, evaluation of the impact of these activities – whether they have moved the needle in the right direction and facilitated formation and functioning of new effective interdisciplinary research teams – have received limited attention [8].…”

“…While several validated measures of team performance are based on direct observation of team activities, these measures are not practical (take a long time to collect data and require large staff and budget) for the assessment of large numbers of teams (e.g., university-wide), especially research teams in post-COVID times that function asynchronously across several settings [8]. Evaluation strategies based on direct observation are also known to introduce observer bias and are more likely to identify outliers.…”

Intellectual Virtues for Interdisciplinary Research: A Consensual Qualitative Analysis

“…In addition, scales were developed to measure partial aspects of interdisciplinarity: the degree of integration of researchers’ work (Masse et al., 2008), the interdisciplinarity of publications (Abramo, D'Angelo, & Zhang, 2018; Bergmann et al., 2017; Karmakar et al., 2020), the research systems (Wang et al., 2017), and so on. More comprehensively, pathways for assessing interdisciplinarity were also presented recently (Laursen, Motzer, & Anderson, 2022).…”

Exploring the evaluation of inter- and transdisciplinary research proposals: Lessons from Dutch research funding reform

“…However, the international literature shows there is still much unclear about how best to evaluate ITDR and there are broader worries about the efficacy of current assessment frameworks to adequately deal with ITDR proposals (e.g. Laursen et al, 2022;McLeish & Strang, 2016). Within this context NWO's Knowledge Platform for Inter-and Transdisciplinary Research 1 has set up a pilot research project aimed at investigating the current state and potential need for modifications of the assessment of ITDR proposals at NWO, and compiling a set of relevant recommendations available from both academic literature and international experience.…”