Research evaluation per discipline: a peer-review method and its outcomes

Abstract

Context & GoalsResearch evaluations are conducted with different goals and at different governance levels. Goals set by the university's research management (e.g. internal quality assessment, definition of spearhead research groups, evaluation of the PhD support and training processes, ex-ante and ex-post project evaluations) may be very different from those set by national authorities (e.g. foundation or justification for university funding, mapping of the global research potential, rationalizing the higher education sector). can be obtained. The method described here fulfills the government requirements towards the university and serves the university's research policy at its typical management levels. On one hand the evaluations are carried out at the scale of a scientific discipline, a granularity well suited for management at the departmental and university levels. On the other hand the subjects of evaluation are the smallest groups in the university's structure, namely research teams, which allows for local remediation at team level and up to the level of the individual researcher.At the time these ex-post evaluations were introduced at the VUB, previous experience in research evaluation within the academic world primarily concerned ex-ante evaluations of project or program proposals (CRE, 2000). The ex-post evaluations procedure described in this paper complemented the Peer reviewAll types of evaluation methods have their own advantages, limitations and typical bias. In view of the goal that was set, the Research Council chose for a qualitative, peer review based method. TheResearch Council considers peer review as the most effective policy instrument for quality improvement and the best way to obtain a thorough and useful view on the research processes and quality. Furthermore, the peer review principle can be applied to all disciplines. This is not always the case for quantitative models, such as bibliometrics, which are always limited to the coverage of the database used (van Raan, 2005;Nederhof, 2006) 3 . Moreover, research process information, which is crucial for remediation purposes, cannot be obtained solely by analysis of output. Output parameters (publication/citation or other) often ignore or underestimate more subtle aspects of research quality and culture, such as the training aspects and international embedding of young researchers, the research culture in which students are immersed, the impact of research on society, economy, culture, government policy, development aid, the scientific literacy of citizens and the non profit sector. The choice for peer review is supported by international studies, showing it is the most cited evaluation method for research quality (Jongbloed et al., 2000; CRE, 2000). The academic community has since long been using peer review (Rigby, 2003). While peer review is generally accepted as the principal method to evaluate research quality, its sensitivity to bias is well known. Critical observations on peer review, performance measures, impacts of evaluations an...

Is the expertise of evaluation panels congruent with the research interests of the research groups: A quantitative approach based on barycenters

“…The focus of these evaluations is research, in particular research quality. Expert panel review is considered the standard for determining research quality of individuals and groups (Nedeva et al, 1996;Rons et al, 2008;Butler & McAllister, 2011;Lawrenz et al, 2012), but also, for instance, for research proposals submitted to research funding organizations (Li & Agha, 2015). In 2007, the University of Antwerp, Belgium, decided to introduce evaluative site visits by expert panels, during which the panel meets the spokesperson of each research group and other relevant stakeholders, and panel members are given the opportunity to ask additional questions or request clarification of specific points described in the self-evaluation report they received in advance.…”

Site Visits: Conversations for Practice

“…The Government Accountability Office (GAO) found that expert panel review and site visits were frequently used in evaluating federal program performance for all federal programs (GAO, ). In terms of validity, expert panel review is viewed as the gold standard for ascertaining research quality and other assessment methods are often compared to it (Butler & McAllister, ; Rons, DeBruyn & Cornelis, ). Generally, this type of site visit is conducted by a team of experts who provide a connoisseurship approach to evaluation (Eisner, ) and follow procedures outlined in Lawrenz, Thao, and Johnson ().…”

Modelos de evaluación del desempeño de las actividades científicas: casos Colombia y México

“…Otros autores plantean modelos de evaluación del desempeño para las actividades científicas en ámbitos específicos como las instituciones públicas (Coccia, 2005), la importancia de que ese proceso se realice con las particularidades de cada disciplina (Benneworth & Jongbloed, 2010;Broadhead & Howard, 1998;Lewison et al, 2007;Rons et al, 2008).…”