ReproducedPapers.org: Openly Teaching and Structuring Machine Learning Reproducibility

Yildiz, Burak; Hung, Hayley; Krijthe, Jesse H.; Liem, Cynthia C. S.; Loog, Marco; Migut, Gosia; Oliehoek, Frans A.; Panichella, Annibale; Pawełczak, Przemysław; Picek, Stjepan; Weerdt, Mathijs de; Gemert, Jan C. van

doi:10.1007/978-3-030-76423-4_1

Cited by 6 publications

(4 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Still, open access is only an aspect of open science, and insights and methods reported in a paper may not trivially be reproducible or replicable 2 , either because common specifications are not sufficiently detailed [9], or because claims may be outright false [10]. While researchers have been divided on which domains suffer from reproducibility crises [11], generally, many well-published works have failed to replicate in psychology [12] and cancer biology [13], and many concerns are arising about the replicability of machine learning outcomes [14], [15]. This leads to credibility crises, in which it is unclear whether results can actually be trusted and built upon.…”

Section: A Insufficient Quality Control On Papersmentioning

confidence: 99%

Treat societally impactful scientific insights as open-source software artifacts

Liem

Demetriou

2023

2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS)

Self Cite

View full text Add to dashboard Cite

So far, the relationship between open science and software engineering expertise has largely focused on the open release of software engineering research insights and reproducible artifacts, in the form of open-access papers, open data, and open-source tools and libraries.In this position paper, we draw attention to another perspective: scientific insight itself is a complex and collaborative artifact under continuous development and in need of continuous quality assurance, and as such, has many parallels to software artifacts. Considering current calls for more open, collaborative and reproducible science; increasing demands for public accountability on matters of scientific integrity and credibility; methodological challenges coming with transdisciplinary science; political and communication tensions when scientific insight on societally relevant topics is to be translated to policy; and struggles to incentivize and reward academics who truly want to move into these directions beyond traditional publishing habits and cultures, we make the parallels between the emerging open science requirements and concepts already well-known in (open-source) software engineering research more explicit. We argue that the societal impact of software engineering expertise can reach far beyond the software engineering research community, and call upon the community members to proactively help driving the necessary systems and cultural changes towards more open and accountable research.

show abstract

Section: A Insufficient Quality Control On Papersmentioning

confidence: 99%

Treat societally impactful scientific insights as open-source software artifacts

Liem

Demetriou

2023

2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In fact, and as we exposed in previous sections, IPOL is a quite useful journal to show the robustness and effectiveness of published scientific methods due to its demo system. Moreover, as pointed out in a recent survey on reproducibility, the educational role of reproducing projects is of main interest both from AI researcher and students' point of view (Yildiz et al, 2021). It is an opportunity to enhance their science contributions, facilitate its understanding and reuse in a way that other members of the scientific community can increase its reach.…”

Section: Ipol As a Tool For Educationmentioning

confidence: 99%

“…The first track was dedicated to the platforms helping for RR research. Different contributions were proposed including for instance the presentation of new libraries like OpenMVG(Moulon et al, 2017) useful for reproducibility or more recently a new platform called ReproducedPapers focusing on reproducibility and openly teaching in machine learning(Yildiz et al, 2021). The second call of RR results was introduced in order to offer the possibility to main ICPR authors (or other past events) to highlight the reproducibility quality of their work including sensibility measure from parameter changes, implementation details and detailed algorithms, links to online demo and source code.…”

mentioning

confidence: 99%

The approach to reproducible research of the Image Processing On Line (IPOL) journal

2022

Informatio

View full text Add to dashboard Cite

Scientific research requires that results can be accessed, tested, and replicated by other researchers. In general, performing reproducible research is not simple and it can be even impossible (for example, in the case of astrophysics replicating singular events if even out of the control of the researcher). However, in computational sciences there is no special hindrance to reproduce, repeat, and compare results. In 2009 the Image Processing On Line (IPOL) journal was founded as a modest contribution to implement reproducible research in the Image Processing field, and then expanded to more general signal-processing algorithms, such as video or physiological signal processing, among others. We re-defined the concept of publication, which is no longer just the article, but the combination of the article, its source code, and any associated data needed to reproduce the results, everything as an indivisible whole. In this article, we present the concept of IPOL, our approach to reproducible research, the challenges we had to address and solve, and some applications other than image processing in IPOL (biomedical, educational). We also present other IPOL inspired initiatives before discussing the opportunity of performing reproducible research in these fields, as well as the advantages and difficulties.

show abstract

“…This increasing interest is also visible through the development of platforms for reproducible research [2]. For instance, since the publication date of this latter review, new major platform appeared like the ReproducedPapers.org platform [3] that allows researchers to share reproduction experience on papers especially in the machine learning field. Such a new platform reaches also educational purpose with, for instance, the integration of reproducibility into fairness, accountability, confidentiality and transparency in artificial intelligence [4].…”

Section: Introductionmentioning

confidence: 99%

Development Efforts for Reproducible Research: Platform, Library and Editorial Investment

Colom,

Hernández,

Kerautret

et al. 2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Reproducible research in pattern recognition can be viewed from a number of angles, including code execution, platforms that promote reproducibility, code sharing, or the release of libraries providing access to relevant algorithms in the corresponding disciplines. In this work, after recalling the motivation and classic definitions of reproducible research, we propose an updated overview of the main platforms that might be used for reproducible research. We then review the different libraries that are commonly used by the pattern recognition, computer vision, imaging and geometry processing communities, and we share our experience of developing a research library. In the third part, new advanced editorial investments will be presented, such as the IPOL journal or other IPOL-inspired new initiatives like OVD-SaaS.

show abstract

ReproducedPapers.org: Openly Teaching and Structuring Machine Learning Reproducibility

Cited by 6 publications

References 15 publications

Treat societally impactful scientific insights as open-source software artifacts

Treat societally impactful scientific insights as open-source software artifacts

The approach to reproducible research of the Image Processing On Line (IPOL) journal

Development Efforts for Reproducible Research: Platform, Library and Editorial Investment

Contact Info

Product

Resources

About