Beyond blind faith: overcoming the obstacles to interdisciplinary evaluation

Abstract

Knowledge about how reviewers serving on interdisciplinary panels produce evaluations that are perceived as fair is especially lacking. This paper draws on 81 interviews with panelists serving on five multidisciplinary fellowship competitions. We identify how peer reviewers define "good" interdisciplinary research proposals, and how they understand the procedures for selecting such proposals. To produce an evaluation they perceive as fair, panelists must respect the primacy of disciplinary sovereignty, deference to expertise and methodological pluralism. These rules ensure the preponderance of the voices of experts over non-experts in interdisciplinary panels. In addition, panelists adopt a range of tactics and strategies designed to make other reviewers who lack such expertise trust that their judgments are disinterested and unbiased.

From a Disciplinary to an Interdisciplinary Design Research: Developing an Integrative Approach for Design

“…This list of questions is carefully drawn from recent research analysing the evaluations of inter-and transdisciplinary projects or proposing relevant questions and criteria (e.g. Bergmann et al 2005;Boix Mansilla 2006;Chen & Rossi 1980;Chen 1996;Defila & Di Giulio 1999;Klein 2008b;Lamont et al 2006;Langfeldt 2006;Laudel & Origgi 2006;Späth 2008;Transdisciplinary Research in Energetics and Cancer (TREC) 2006;Wiesmann et al 2008). The selected questions clarify all the dimensions for consideration, emphasises the quality of synthesis and integration, and is divided into five assessment sections: broadness; integration; reflection and learning; problem-solving; and management, social and leadership skills.…”

Supporting Frontier Research, Which Institutions and Which Processes

“…Looking at interdisciplinary grant committees, Lamont et al (2006) propose a more nuanced answer. Analyzing how committees build their criteria and rules, they show that "procedural fairness" is warranted on "respecting disciplinary sovereignty", that this drives to recognising and accepting different "epistemological styles", but that this also allows reviewers to have "their tastes and idiosyncrasies … play a greater role": as mentioned by one of their interviewees "excellence is in some ways what looks most like you".…”

How mechanisms implicated in the collective process of academic hiring in France affect discrimination

“…Results are quite different when a more diverse group of peers are doing the evaluation. Studying committees in charge of selecting grant projects to be funded by American research agencies, Michèle Lamont and her colleagues (Guetzkow et al, 2002;Guetzkow et al, 2004;Lamont et al, 2006) have shown that these committees-pluridisciplinary and therefore diverse; also made up of members who do not know each very well if at all-construct and use transdisciplinary "bridges" between members' distinct epistemological codes and produce more highly diversified results.…”