Detection of data fabrication using statistical tools

Abstract

Scientific misconduct potentially invalidates findings in many scientific fields. Improved detection of unethical practices like data fabrication is considered to deter such practices. In two studies, we investigated the diagnostic performance of various statistical methods to detect fabricated quantitative data from psychological research. In Study 1, we tested the validity of statistical methods to detect fabricated data at the study level using summary statistics. Using (arguably) genuine data from the Many Labs 1 project on the anchoring effect (k=36) and fabricated data for the same effect by our participants (k=39), we tested the validity of our newly proposed 'reversed Fisher method', variance analyses, and extreme effect sizes, and a combination of these three indicators using the original Fisher method. Results indicate that the variance analyses perform fairly well when the homogeneity of population variances is accounted for and that extreme effect sizes perform similarly well in distinguishing genuine from fabricated data. The performance of the 'reversed Fisher method' was poor and depended on the types of tests included. In Study 2, we tested the validity of statistical methods to detect fabricated data using raw data. Using (arguably) genuine data from the Many Labs 3 project on the classic Stroop task (k=21) and fabricated data for the same effect by our participants (k=28), we investigated the performance of digit analyses, variance analyses, multivariate associations, and extreme effect sizes, and a combination of these four methods using the original Fisher method. Results indicate that variance analyses, extreme effect sizes, and multivariate associations perform fairly well to excellent in detecting fabricated data using raw data, while digit analyses perform at chance levels. The two studies provide mixed results on how the use of random number generators affects the detection of data fabrication. Ultimately, we consider the variance analyses, effect sizes, and multivariate associations valuable tools to detect potential data anomalies in empirical (summary or raw) data. However, we argue against widespread (possible automatic) application of these tools, because some fabricated data may be irregular in one aspect but not in another. Considering how violations of the assumptions of fabrication detection methods may yield high false positive or false negative probabilities, we recommend comparing potentially fabricated data to genuine data on the same topic.

Integrity culture is underpinned by education, not post-submission dishonesty assessments

“…Early postsubmission honesty assessment is flawed because current tools for 'after-thefact' integrity checking have generally not been validated (Khan, 2021b). In a robust evaluation comparing integral versus fabricated data (Hartgerink et al, 2019), the authors' concluded: 'we argue against widespread (possible automatic) application of these tools'. Instead of targeting submissions for early integrity assessments with doubtful tools, journals need to put their own house in order, for example by first getting their instructions to authors right regarding transparency (Malički et al, 2021).…”

The Four Prominent Indicators of Diagnostic Testing: Consistency among their Values Reported in Saudi Medical Journals

“…The findings of Carlisle have ajor i plications and ra ifications for edical science and its publication systems [57]. The (mainly statistical) methods available for detecting false data, which are devised by Carlisle and others [58][59][60][61][62][63][64][65][66], have become rather visible and advanced during the past decade. Rushdi and Rushdi [4] have recently suggested a non-statistical method based on the premise that flawed data might be detected via the excessive inconsistencies it causes in a variant of Boolean Analysis called Qualitative Comparative Analysis (QCA) [4,[67][68][69][70][71][72][73][74][75][76].…”

“…Moreover, it pertains to the validity of the premise that medical science is up to its claim of being self-correcting. The literature abounds with excellent articles related to this topic [51][52][53][54][55][56][57][58][59][60][61][62][63][64][65][66][77][78][79][80][81][82][83][84][85][86][87][88].…”

Has the Pandemic Triggered a ‘Paperdemic’? Towards an Assessment of Diagnostic Indicators for COVID-19

“…Even before the COVID-19 era, there have been many methods (mainly statistical) for detecting false data [27][28][29][30][31][32][33][34][35][36][37], that have become rather visible and advanced during the past decade. Rushdi and Rushdi [38,39] have recently suggested methods for avoiding probabilistic fallacies in medical context.…”