Fraudulent studies are undermining the reliability of systematic reviews – a study of the prevalence of problematic images in preclinical studies of depression

Abstract

Systematic reviews are considered by many to constitute the highest level of scientific evidence. A caveat is that the methods used in a systematic review - combining information from multiple studies - are predicated on all of the reports being truthful. Currently, we do not know how frequent fraudulent studies are in systematic reviews, or how they affect the resulting evidence base. For a systematic review of preclinical studies of depression, we found that potentially fraudulent studies were not only common but also that they biased the findings of the review. In a sample of 1,035 studies, we found that 19 % of peer-reviewed reports displayed data in the form of problematic images. In a majority of the cases, images had been altered or recycled in a way that makes us suspect foul play. Making things worse, these studies reported larger effect sizes, on average, than did studies where we did not identify problems. Counter to commonly held beliefs, reports with problematic images were not cited less or published in lower-impact journals, nor were their authors isolated to any specific geographic area. The sheer prevalence of problematic studies, and the fact that we could not find a simple pattern for identifying them, undermines the validity of systematic reviews within our research field. We suspect that this is symptomatic of a broader problem that needs immediate addressing.