Responsible Research Assessment I: Implementing DORA for hiring and promotion in psychology

Abstract

The use of journal impact factors and other metric indicators of research productivity, such as the h-index, has been heavily criticized for being invalid for the assessment of individual researchers and for fueling a detrimental “publish or perish” culture. Multiple initiatives call for developing alternatives to existing metrics that better reflect quality (instead of quantity) in research assessment. This report, written by a task force established by the German Psychological Society, proposes how responsible research assessment could be done in the field of psychology. We present four principles of responsible research assessment in hiring and promotion and suggest a two-phase assessment procedure that combines the objectivity and efficiency of indicators with a qualitative, discursive assessment of shortlisted candidates. The main aspects of our proposal are (a) to broaden the range of relevant research contributions to include published data sets and research software, along with research papers, and (b) to place greater emphasis on quality and rigor in research evaluation.

Responsible Research Assessment II: A specific proposal for hiring and promotion in psychology

“…Needless to say, there are other types of academic contributions, such as teaching, leadership, service to the academic institution/field, and societal impact (cf. Schönbrodt et al, 2022), which should be equally considered during hiring and promotion decisions. The evaluation scheme proposed here focuses on research outputs, but we encourage the scientific community to develop criteria for the other domains as well, to further enhance fairness, transparency and standardization in the hiring and promotion process.…”

“…These should then be rated according to the quality criteria listed in Table 1. The criteria are based on Leising et al (2022) with some modifications (e.g., no evaluation of scientific consensus; further specification of credit roles, FAIR format of data/scripts; narrative statements on statistical power, sample size and impact). The rationale behind this is to make visible the average methodological quality of the applicant's best, most current work, while also largely eliminating mere quantity as an assessment criterion.…”

“…There is now a wide-spread demand for specified alternatives to current academic evaluation practices. In a previous report, we laid out four basic principles of a more responsible research assessment in academic hiring and promotion processes (Schönbrodt et al, 2022). The present paper offers a specific proposal for how these principles may be implemented in practice: We argue in favor of broadening the range of relevant research contributions and thus propose concrete quality criteria (including ready-to-use online templates) for published research articles, data sets and research software.…”

“…However, it is still unclear how exactly a "better" approach would look like. Schönbrodt et al (2022) proposed four principles for a more quality-based research assessment in psychological science: (1) Academic contributions are multifaceted. Regarding research contributions, do not only value (a) journal articles, but also (b) data sets and (c) research software development.…”

“…Phase 1: Establishing a minimum threshold for methodological rigor Schönbrodt et al (2022) proposed assessing the academic achievements and potential of applicants in two consecutive phases: In Phase 1, the overall methodological rigor of an applicant's research is assessed by assigning quality scores to individual pieces of research output to which the candidate has contributed. In order to keep the workload manageable for committees, this assessment has algorithmic elements and uses indicators that are relatively objective and easy to obtain.…”