Peer reviewer’s dilemmas: a qualitative exploration of decisional conflict in the evaluation of grant applications in the medical humanities and social sciences

Abstract

Independent evaluations of grant applications by subject experts are an important part of the peer-review system. However, little is known about the real-time experiences of peer reviewers or experts who perform reviews of a grant application independently. This study sought to gain insight into this stage of the grant evaluation process by observing how experts conduct an independent review in near real time. Using the think aloud approach and Critical Decision Method of interviewing, in-depth interviews were conducted with 16 peer reviewers from a range of roles and disciplines within the medical humanities and social sciences. Participants were asked to think aloud while reviewing applications to different grant schemes from a single prestigious funder. The analysis shows reviewers encountered five dilemmas during the evaluation process. These dilemmas were related to whether or not one should (1) accept an invitation to review, (2) rely exclusively on the information presented in the application, (3) pay attention to institutional prestige, (4) offer comments about aspects that are not directly related to academics’ area of expertise, and (5) to take risks and overlook shortcomings rather than err on the side of caution. In order to decide on the appropriate course of action, reviewers often engaged in a series of deliberations and trade-offs – varying in length and complexity. However, their interpretation of what was ‘right’ was influenced by their values, preferences and experiences, but also by relevant norms and their understanding of the funder’s guidelines and priorities. As a result, the way reviewers approached the identified dilemmas was idiosyncratic and sometimes diametrically opposed to other reviewers’ views, which could lead to variation in peer-review outcomes. The dilemmas we have uncovered suggest that peer reviewers engage in thoughtful considerations during the peer-review process. We should, therefore, be wary of reducing the absence of consensus as resulting from biased, instinctive thinking. Rather, these findings highlight the diversity of values, priorities and habits and ways of working each reviewer brings to the fore when reviewing the applicants and their project proposals and call for further reflection on, and study of, this “invisible work” to better understand and continue to improve the peer-reviewing process.

Targeted, actionable and fair: Reviewer reports as feedback and its effect on ECR career choices

“…From the reviewers’ perspectives, wide ranging culture change is needed to ensure that reviewers are aware that their comments is used as feedback by applicants and is not solely, as is currently understood ( Vallée-Tourangeau et al 2022 ), as a binary selection mechanism (successful/unsuccessful). Instead, reviewer training should highlight that grant-peer review involves two objectives; (1) to assist the decision-making process (selection); and (2) to provide comments that can be utilized effectively by applicants as feedback (participation/development).…”

“…Instead, reviewer training should highlight that grant-peer review involves two objectives; (1) to assist the decision-making process (selection); and (2) to provide comments that can be utilized effectively by applicants as feedback (participation/development). This may require large scale changes to how researchers who act as peer reviews currently see their role of which previous research has seen as a series of dilemmas between self-interest (supporting familiar applications) and evaluation (rigorously assessing the proposal) ( Lamont 2009 ; Vallée-Tourangeau et al 2022 ). Whereas the dilemma surrounding self-interest driven reviews ( Lamont 2009 ) may include a developmental consideration, there is still a lack of awareness of how good review commentary can act towards increasingly the likelihood of future success and, conversely the career and personal consequences of reviews that are ill-targeted or else, parsimonious.…”

Targeted, actionable and fair: reviewer reports as feedback and its effect on ECR career choices

“…From the reviewers' perspectives, wide ranging culture change is needed to ensure that reviewers are aware that their comments will be used as feedback and not, as is currently understood (Vallée-Tourangeau et al, 2022), to assist the selection of successful and unsuccessful applicants. Therefore, reviewer training should highlight that grant-peer review involves two objectives; (1) to assist the decision-making process (selection); and (2) to provide comments that can be utilised effectively by applicants as feedback (participation/development).…”

“…Therefore, reviewer training should highlight that grant-peer review involves two objectives; (1) to assist the decision-making process (selection); and (2) to provide comments that can be utilised effectively by applicants as feedback (participation/development). This may require large scale changes to how researchers who act as peer reviews currently see their role of which previous research has seen as a series of dilemmas between selfinterest (supporting familiar applications) and evaluation (rigorously assessing the proposal) (Lamont, 2009;Vallée-Tourangeau et al, 2022). Whereas the dilemma surrounding self-interest driven reviews (Lamont, 2009) may include a developmental consideration, there is still a lack of awareness of how good review commentary can act towards increasingly the likelihood of future success and, conversely the career and personal consequences of reviews that are ill-targeted or else, parsimonious.…”

Targeted, actionable and fair: reviewer reports as feedback and its effect on ECR career choices

“…Instead, reviewer training should highlight that grant-peer review involves two objectives; (1) to assist the decision-making process (selection); and (2) to provide comments that can be utilised effectively by applicants as feedback (participation/development). This may require large scale changes to how researchers who act as peer reviews currently see their role of which previous research has seen as a series of dilemmas between self-interest (supporting familiar applications) and evaluation (rigorously assessing the proposal) (Lamont, 2009;Vallée-Tourangeau et al, 2022). Whereas the dilemma surrounding self-interest driven reviews (Lamont, 2009) may include a developmental consideration, there is still a lack of awareness of how good review commentary can act towards increasingly the likelihood of future success and, conversely the career and personal consequences of reviews that are ill-targeted or else, parsimonious.…”