Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

Barnett, Adrian

doi:10.21203/rs.3.rs-1496802/v1

Cited by 3 publications

(5 citation statements)

References 34 publications

(45 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, an investigator might test the hypothesis that a researcher has fabricated clinical trial data for two supposedly randomised trial groups by assessing the under-or over-dispersion of the summary statistics. Indeed Barnett [55] provides a comprehensive analysis of such a test's effectiveness, concluding that it can be a useful flag of suspect clinical trials in targeted checks. It might reasonably also be applied to the statistics of other between-groups experimental data.…”

Section: Discussionmentioning

confidence: 99%

Investigating and preventing scientific misconduct using Benford’s Law

Eckhartt

Ruxton

2023

Res Integr Peer Rev

View full text Add to dashboard Cite

Integrity and trust in that integrity are fundamental to academic research. However, procedures for monitoring the trustworthiness of research, and for investigating cases where concern about possible data fraud have been raised are not well established. Here we suggest a practical approach for the investigation of work suspected of fraudulent data manipulation using Benford’s Law. This should be of value to both individual peer-reviewers and academic institutions and journals. In this, we draw inspiration from well-established practices of financial auditing. We provide synthesis of the literature on tests of adherence to Benford’s Law, culminating in advice of a single initial test for digits in each position of numerical strings within a dataset. We also recommend further tests which may prove useful in the event that specific hypotheses regarding the nature of data manipulation can be justified. Importantly, our advice differs from the most common current implementations of tests of Benford’s Law. Furthermore, we apply the approach to previously-published data, highlighting the efficacy of these tests in detecting known irregularities. Finally, we discuss the results of these tests, with reference to their strengths and limitations.

show abstract

Section: Discussionmentioning

confidence: 99%

Investigating and preventing scientific misconduct using Benford’s Law

Eckhartt

Ruxton

2023

Res Integr Peer Rev

View full text Add to dashboard Cite

show abstract

“…(16) Are any of the baseline data excessively similar between randomized groups? (17,18) Are any of the baseline data excessively different between randomised groups? (17,18) Are there any discrepancies between the values for percentage and absolute change?…”

Section: Domain 2: Inspecting the Research Team And Their Work (19 Ch...mentioning

confidence: 99%

“…(17,18) Are any of the baseline data excessively different between randomised groups? (17,18) Are there any discrepancies between the values for percentage and absolute change? (7) Are there any discrepancies between reported data and participant inclusion criteria?…”

Section: Domain 2: Inspecting the Research Team And Their Work (19 Ch...mentioning

confidence: 99%

A survey of experts to identify methods to detect problematic studies: Stage 1 of the INSPECT-SR Project

Wilkinson,

Heal,

Antoniou

et al. 2024

Preprint

View full text Add to dashboard Cite

Background Randomised controlled trials (RCTs) inform healthcare decisions. Unfortunately, some published RCTs contain false data, and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesise all RCTs which have been conducted on a given topic. This means that any of these 'problematic studies' are likely to be included, but there are no agreed methods for identifying them. The INSPECT-SR project is developing a tool to identify problematic RCTs in systematic reviews of healthcare-related interventions. The tool will guide the user through a series of `checks' to determine a study's authenticity. The first objective in the development process is to assemble a comprehensive list of checks to consider for inclusion. Methods We assembled an initial list of checks for assessing the authenticity of research studies, with no restriction to RCTs, and categorised these into five domains: Inspecting results in the paper; Inspecting the research team; Inspecting conduct, governance, and transparency; Inspecting text and publication details; Inspecting the individual participant data. We implemented this list as an online survey, and invited people with expertise and experience of assessing potentially problematic studies to participate through professional networks and online forums. Participants were invited to provide feedback on the checks on the list, and were asked to describe any additional checks they knew of, which were not featured in the list. Results Extensive feedback on an initial list of 102 checks was provided by 71 participants based in 16 countries across five continents. Fourteen new checks were proposed across the five domains, and suggestions were made to reword checks on the initial list. An updated list of checks was constructed, comprising 116 checks. Many participants expressed a lack of familiarity with statistical checks, and emphasized the importance of feasibility of the tool. Conclusions A comprehensive list of trustworthiness checks has been produced. The checks will be evaluated to determine which should be included in the INSPECT-SR tool.

show abstract

“…The steps are outlined below and the complete code is available on GitHub. 24 I downloaded a list of published randomised trials from the trialstreamer web page 25 using the PubMed Central ID (PMCID). The trialstreamer data was downloaded on 9 August 2021 and had 57,109 trials with a PMCID.…”

Section: Automated Extraction Of Baseline Tablesmentioning

confidence: 99%

“…26 Estimation All the R code to extract the tables and run the Bayesian model is openly available https://github.com/agbarnett/ baseline_tables. 24 An interactive version of my Bayesian model is available via shiny: https://aushsi.shinyapps.io/ baseline/. The Bayesian models were fitted using WinBUGS Version 1.4.3 27 for the paper and nimble version 0.12.1 28 for the shiny application.…”

Section: Predictors Of Under-or Over-dispersionmentioning

confidence: 99%

Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

Barnett

2023

F1000Res

Self Cite

View full text Add to dashboard Cite

Background: Papers describing the results of a randomised trial should include a baseline table that compares the characteristics of randomised groups. Researchers who fraudulently generate trials often unwittingly create baseline tables that are implausibly similar (under-dispersed) or have large differences between groups (over-dispersed). I aimed to create an automated algorithm to screen for under- and over-dispersion in the baseline tables of randomised trials. Methods: Using a cross-sectional study I examined 2,245 randomised controlled trials published in health and medical journals on PubMed Central. I estimated the probability that a trial's baseline summary statistics were under- or over-dispersed using a Bayesian model that examined the distribution of t-statistics for the between-group differences, and compared this with an expected distribution without dispersion. I used a simulation study to test the ability of the model to find under- or over-dispersion and compared its performance with an existing test of dispersion based on a uniform test of p-values. My model combined categorical and continuous summary statistics, whereas the uniform test used only continuous statistics. Results: The algorithm had a relatively good accuracy for extracting the data from baseline tables, matching well on the size of the tables and sample size. Using t-statistics in the Bayesian model out-performed the uniform test of p-values, which had many false positives for skewed, categorical and rounded data that were not under- or over-dispersed. For trials published on PubMed Central, some tables appeared under- or over-dispersed because they had an atypical presentation or had reporting errors. Some trials flagged as under-dispersed had groups with strikingly similar summary statistics. Conclusions: Automated screening for fraud of all submitted trials is challenging due to the widely varying presentation of baseline tables. The Bayesian model could be useful in targeted checks of suspected trials or authors.

show abstract

Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

Cited by 3 publications

References 34 publications

Investigating and preventing scientific misconduct using Benford’s Law

Investigating and preventing scientific misconduct using Benford’s Law

A survey of experts to identify methods to detect problematic studies: Stage 1 of the INSPECT-SR Project

Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

Contact Info

Product

Resources

About