A Framework for the Assessment of Research and Its Impacts

Abstract

This paper proposes a holistic framework for the development of models for the assessment of research activities and their impacts. It distinguishes three dimensions, including in an original way, data as a main dimension, together with theory and methodology. Each dimension of the framework is further characterized by three main building blocks: education, research, and innovation (theory); efficiency, effectiveness, and impact (methodology); and availability, interoperability, and “unit-free” property (data). The different dimensions and their nine constituent building blocks are attributes of an overarching concept, denoted as “quality.” Three additional quality attributes are identified as implementation factors (tailorability, transparency, and openness) and three “enabling” conditions (convergence, mixed methods, and knowledge infrastructures) complete the framework. A framework is required to develop models of metrics. Models of metrics are necessary to assess the meaning, validity, and robustness of metrics. The proposed framework can be a useful reference for the development of the ethics of research evaluation. It can act as a common denominator for different analytical levels and relevant aspects and is able to embrace many different and heterogeneous streams of literature. Directions for future research are provided.

A Tailor-made Data Quality Approach for Higher Educational Data

“…The development of models of indicators or metrics for a quantitative assessment requires a comprehensive framework, which should include the specification of the underlying theory, methodology, and data properties. Models of metrics are necessary to assess the meaning, validity, and robustness of metrics (Daraio, 2017). Daraio and Glanzel (2016) identified the following critical issues: i) data quality issues (OECD, 2011) including completeness, validity, accuracy, consistency, availability, and timeliness; ii) comparability problems related to heterogeneous definitions of the variables, data collection practices, and databases; iii) lack of standardization; iv) lack of interoperability; v) lack of modularization; vi) problems of classification; vii) difficulties in the creation of concordance tables among different classification schemes; viii) problems and costs of the extensibility of the system; ix) problems and costs of updating of the system.…”

Performance model’s development: a novel approach encompassing ontology-based data access and visual analytics

“…The main components of a performance evaluation model can be found in Daraio (2017Daraio ( , 2019. We have actors that are involved in processes which consist in the combination and or transformation of inputs in outputs, taking into account the main objectives of the activities.…”

“…These changes produce different effects (see further details and references in Daraio 2019, Table 24.2, p. 644) (i) on the demand side (those that ask for research assessment) including an increase of institutional and internal assessments, (ii) on the supply side (those that offer research assessment) including proliferation of rankings, development of Altmetrics, open access repositories, new assessment tools and desktop bibliometrics), (iii) on scholars (the increase of "publish or perish" pressure, impact on the incentives, behaviour and misconduct, and increasing critics against traditional bibliometric indicators), (iv) on the assessment process (increasing the complexity of the research assessment) and on the indicators' development. Daraio (2017) showed that the formulation of models (in this paper we will use metrics and indicators interchangeably) is necessary to assess the meaning, validity and robustness of metrics. It was observed that developing models is important for learning about the explicit consequences of assumptions, for testing the assumptions, for documenting and verifying the assumptions, for systematizing the problem and the choices done.…”

“…Little attention is paid to the problem of developing performance indicators. As highlighted in Daraio (2017Daraio ( , 2019 it is necessary to describe the theoretical, methodological and data components that constitute the model of a metric in order to evaluate its appropriateness. Without a reference model, it is not possible to evaluate the robustness of the metric used in a performance evaluation.…”

Policy turnaround: Towards a new deal for research and higher education. Governance, evaluation and rankings in the big data era

“…Daraio and Bonaccorsi () illustrate the design of a possible information system to integrate microdata on universities with other sources with the aim of overcoming rankings by linking data in an open platform. In particular, the new developments may be useful to consider data quality dimensions and the openness of the data platform, although the limits of data and their availability and interoperability still remain a critical issue (Borgman, ; Daraio, Lenzerini, Leporelli, Naggar, et al, ) that need to be considered in the development of metrics for the assessment of research, education and innovation (Daraio, ).…”