Issues with data and analyses: Errors, underlying themes, and potential solutions

Abstract

Some aspects of science, taken at the broadest level, are universal in empirical research. These include collecting, analyzing, and reporting data. In each of these aspects, errors can and do occur. In this work, we first discuss the importance of focusing on statistical and data errors to continually improve the practice of science. We then describe underlying themes of the types of errors and postulate contributing factors. To do so, we describe a case series of relatively severe data and statistical errors coupled with surveys of some types of errors to better characterize the magnitude, frequency, and trends. Having examined these errors, we then discuss the consequences of specific errors or classes of errors. Finally, given the extracted themes, we discuss methodological, cultural, and systemlevel approaches to reducing the frequency of commonly observed errors. These approaches will plausibly contribute to the self-critical, self-correcting, ever-evolving practice of science, and ultimately to furthering knowledge.rigor | statistical errors | quality control | reproducibility | data analysis

Toward fulfilling the aspirational goal of science as self‐correcting: A call for editorial courage and diligence for error correction

“…3 Such errors might arise from honest error, poor research practices, or misconduct. In such cases, scholarly discussion among authors and readers is healthy and may not require editorial facilitation, intervention, or correction of the original article.…”

“…Throughout this paper, we focus on unequivocal errors that involve factual or analytical mistakes, some of which, if corrected, might alter a study's conclusions. 3 Such errors might arise from honest error, poor research practices, or misconduct.…”

New Guidelines for Data Reporting and Statistical Analysis: Helping Authors With Transparency and Rigor in Research

“…Statistical models are powerful methods to filter signals from noise in experimental data and help make accurate inferences. However, the application of statistics has been beset by misuse in biomedical research; following is a brief discussion of the most frequent problems …”

“…However, the application of statistics has been beset by misuse in biomedical research; (14) following is a brief discussion of the most frequent problems. (15,16) Overfitting is a major threat to the validity and reproducibility of multivariable statistical models. In regression analyses, overfitting occurs when a model with too many parameters is fitted to a small data set, such that the number of parameters is greater than the number of events.…”

Rigor, reproducibility, and novel methodological approaches to eating disorders research

“…To the best of our knowledge, there are no formal scoring criteria for rigor and reproducibility, although they are often incorporated into peer review guidelines and evaluation criteria. As the call for a cultural shift towards prioritizing methodological rigor and reporting continues (Brown, Kaiser, & Allison, ), peer reviewers must hold some responsibility for developing and applying these core scientific constructs. The future will undoubtedly offer some quantification of these dimensions in publications or proposals under review, but we hope that the contents of this issue motivate International Journal of Eating Disorders reviewers to more fully consider rigor and reproducibility in their peer review.…”