On the evaluation of research software: the CDUR procedure

Abstract

Background: Evaluation of the quality of research software is a challenging and relevant issue, still not sufficiently addressed by the scientific community. Methods: Our contribution begins by defining, precisely but widely enough, the notions of research software and of its authors followed by a study of the evaluation issues, as the basis for the proposition of a sound assessment protocol: the CDUR procedure. Results: CDUR comprises four steps introduced as follows: Citation, to deal with correct RS identification, Dissemination, to measure good dissemination practices, Use, devoted to the evaluation of usability aspects, and Research, to assess the impact of the scientific work. Conclusions: Some conclusions and recommendations are finally included. The evaluation of research is the keystone to boost the evolution of the Open Science policies and practices. It is as well our belief that research software evaluation is a fundamental step to induce better research software practices and, thus, a step towards more efficient science.

A Manifesto on Resource Re-Use in Interactive Information Retrieval

“…Even for more generic aspects of the IIR research infrastructure, such as software to manage the participants flow through an experiment, the attempts at developing a re-usable solution [25,54,57] have not achieved any uptake. This mirrors the state of research software re-use in general [2,14,21].…”

“…Second, 'free' and long-term options such as GitHub Pages are widely available, ensuring at least medium-term availability. Since research software is rarely stable and well-documented [22], this is may be an even bigger challenge then documenting and archiving research designs and research data, as evident from the lack of re-use of IIR research software [25,54,57] and research software in general [2,14,21].…”

An environment for sustainable research software in Germany and beyond: current state, open challenges, and call for action

“…Several evaluation schemes for research software have been proposed before and led to the formulation of first recommendations 13 , 14 . Gomez-Diaz & Recio suggested the CDUR scheme based on Citation, Dissemination (including aspects like license, web site, contact point), Use, and Research (output) 61 . Lamprecht et al .…”

Open comments on the Task Force SIRS report: Scholarly Infrastructures for Research Software (EOSC Executive Board, EOSCArchitecture)

“…Should EOSC infrastructures ask to private companies to produce references, metadata, and links to related research articles and research data for their produced software? Please note that the CDUR procedure to evaluate research software has been proposed in (Gomez-Diaz and Recio 2019). CDUR comprises four steps introduced as follows: Citation, to deal with correct RS identification, Dissemination, to measure good dissemination practices, Use, devoted to the evaluation of usability aspects, and Research, to assess the impact of the scientific work.…”