The limits of replicability

Güttinger, Stephan

doi:10.1007/s13194-019-0269-1

Cited by 35 publications

(39 citation statements)

References 51 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…As I have argued, the contextual approach suggests to move away from viewing quality in universal terms and in the direction of locality and dependency, also in the context of the current debate on reproducibility. At the same time, similarly to the previous discussion on re-use, this brings up the question of how and when to apply reproducibility standards [43]. Here, the contextual approach follows a "local" approach to the issue, according to which requiring reproducibility as a general standard is highly problematic and discussing the issue on a case-by-case basis is the direction to follow.…”

Section: The Contextual Approach In Practicementioning

confidence: 89%

Towards a Contextual Approach to Data Quality

Canali

2020

Data

View full text Add to dashboard Cite

In this commentary, I propose a framework for thinking about data quality in the context of scientific research. I start by analyzing conceptualizations of quality as a property of information, evidence and data and reviewing research in the philosophy of information, the philosophy of science and the philosophy of biomedicine. I identify a push for purpose dependency as one of the main results of this review. On this basis, I present a contextual approach to data quality in scientific research, whereby the quality of a dataset is dependent on the context of use of the dataset as much as the dataset itself. I exemplify the approach by discussing current critiques and debates of scientific quality, thus showcasing how data quality can be approached contextually.

show abstract

Section: The Contextual Approach In Practicementioning

confidence: 89%

Towards a Contextual Approach to Data Quality

Canali

2020

Data

View full text Add to dashboard Cite

show abstract

“…Lowering the bar on the number of (control) experiments conducted or data sets included can have multiple implications for the quality of the science conducted. For several decades now, multiple scholars have claimed science to be in a ‘reproducibility crisis’ ( Guttinger 2020 ; Nelson et al. 2020 ).…”

Section: Discussionmentioning

confidence: 99%

No time for that now! Qualitative changes in manuscript peer review during the Covid-19 pandemic

Horbach

2021

Research Evaluation

View full text Add to dashboard Cite

The global Covid-19 pandemic has had considerable impact on the scientific enterprise, including scholarly publication and peer review practices. Several studies have assessed these impacts, showing among others that medical journals have strongly accelerated their review processes for Covid-19 related content. This has raised questions and concerns regarding the quality of the review process and the standards to which manuscripts are held for publication. To address these questions, this study sets out to assess qualitative differences in review reports and editorial decision letters for Covid-19 related, articles not related to Covid-19 published during the 2020 pandemic, and articles published before the pandemic. It employs the open peer review model at the British Medical Journal and eLife to study the content of review reports, editorial decisions, author responses, and open reader comments. It finds no clear differences between review processes of articles not related to Covid-19 published during or before the pandemic. However, it does find notable diversity between Covid-19 and non-Covid-19 related articles, including fewer requests for additional experiments, more cooperative comments, and different suggestions to address too strong claims. In general, the findings suggest that both reviewers and journal editors implicitly and explicitly use different quality criteria to assess Covid-19 related manuscripts, hence transforming science’s main evaluation mechanism for their underlying studies and potentially affecting their public dissemination.

show abstract

“…Finally, replication studies are generally challenging [109]; the largest replication study to date attempted to replicate 100 studies and succeeded only in 39% of them [14]. Human test subjects consciously or sub-consciously remember previous experiences that can impact their thoughts, behaviour, and performance; thus, experiments can result in different results due to the non-uniformity of nature [35,90]. While the original work was published in 2018 [52], our studies were conducted two years later during a pandemic [77] -which has to be noted.…”

Section: Limitationsmentioning

confidence: 99%

RepliCueAuth: Validating the Use of a Lab-Based Virtual Reality Setup for Evaluating Authentication Systems

Mathis

Vaniea

Khamis

2021

Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

View full text Add to dashboard Cite

Figure 1: We evaluate and discuss the suitability of using Virtual Reality (VR) to conduct human-centred usability and security evaluations of real-world authentication systems. To this end, we replicated a recently introduced authentication scheme called CueAuth [52] (➊) into VR (➋). We then evaluate the usability and security of our replica and compare the results to the real-world evaluation of CueAuth [52].

show abstract

The limits of replicability

Cited by 35 publications

References 51 publications

Towards a Contextual Approach to Data Quality

Towards a Contextual Approach to Data Quality

No time for that now! Qualitative changes in manuscript peer review during the Covid-19 pandemic

RepliCueAuth: Validating the Use of a Lab-Based Virtual Reality Setup for Evaluating Authentication Systems

Contact Info

Product

Resources

About