Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions

Wilkinson, Jack; Heal, Calvin; Antoniou, George A.; Flemyng, Ella; Alfirevic, Zarko; Avenell, Alison; Barbour, Virginia; Brown, Nicholas J L; Carlisle, John; Clarke, Mike; Dicker, Patrick; Dumville, Jo; Grey, Andrew; Grohmann, Steph; Gurrin, Lyle C; Hayden, Jill A; Heathers, James; Hunter, Kylie E; Lasserson, Toby; Lam, Emily; Lensen, Sarah; Li, Tianjing; Li, Wentao; Loder, Elizabeth; Lundh, Andreas; Meyerowitz-Katz, Gideon; Mol, Ben W; O’Connell, Neil E; Parker, Lisa; Redman, Barbara K.; Seidler, Anna Lene; Sheldrick, Kyle A; Sydenham, Emma; Torgerson, David J; van Wely, Madelon; Wang, Rui; Bero, Lisa; Kirkham, Jamie J

doi:10.1101/2023.09.21.23295626

Cited by 4 publications

(7 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We conducted an international survey of experts to elaborate an extensive list of potential checks for identifying problematic studies. The items on the list will be evaluated for their usefulness and feasibility to determine which checks should be included in the INSPECT-SR tool and any implications for the tool’s structure (3). It should be emphasised that a check’s inclusion on the list does not amount to an endorsement by the research team.…”

Section: Discussionmentioning

confidence: 99%

“…The items on the list will be evaluated via an application of the items on the list to RCTs in 50 Cochrane Systematic Reviews, an online Delphi survey, and consensus meetings, to produce a draft version of the INSPECT-SR tool. The draft version will then be subject to testing by users, and feedback from this testing will be used to improve and finalize the tool (3). The final version will represent a feasible tool, backed by empirical evidence and broad expert consensus, for evaluating potentially problematic studies in health-related systematic reviews.…”

Section: Discussionmentioning

confidence: 99%

“…The domains used were Inspecting results in the paper , Inspecting the research team , Inspecting conduct, governance and transparency, Inspecting text and publication details , and Inspecting individual participant data . The wording and categorization of the checks was reviewed by the project Expert Panel (3) and revised accordingly. The majority were rephrased as questions for consistency.…”

Section: Methodsmentioning

confidence: 99%

“…The methods used in this study have been described in an online protocol (https://osf.io/6pmx5/) and in a protocol paper describing the INSPECT-SR project (3). We give an overview here.…”

Section: Methodsmentioning

confidence: 99%

“…This prompts the question of how we can systematically detect problematic studies. The overall aim of the INSPECT-SR (INveStigating ProblEmatic Clinical Trials in Systematic Reviews) project is to develop and evaluate a tool for identifying problematic studies in the context of systematic reviews of RCTs of health interventions(3). The INSPECT-SR tool will guide the user through a series of ‘checks’ for study trustworthiness.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

A survey of experts to identify methods to detect problematic studies: Stage 1 of the INSPECT-SR Project

Wilkinson,

Heal,

Antoniou

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

Background Randomised controlled trials (RCTs) inform healthcare decisions. Unfortunately, some published RCTs contain false data, and some appear to have been entirely fabricated. Systematic reviews are performed to identify and synthesise all RCTs which have been conducted on a given topic. This means that any of these 'problematic studies' are likely to be included, but there are no agreed methods for identifying them. The INSPECT-SR project is developing a tool to identify problematic RCTs in systematic reviews of healthcare-related interventions. The tool will guide the user through a series of `checks' to determine a study's authenticity. The first objective in the development process is to assemble a comprehensive list of checks to consider for inclusion. Methods We assembled an initial list of checks for assessing the authenticity of research studies, with no restriction to RCTs, and categorised these into five domains: Inspecting results in the paper; Inspecting the research team; Inspecting conduct, governance, and transparency; Inspecting text and publication details; Inspecting the individual participant data. We implemented this list as an online survey, and invited people with expertise and experience of assessing potentially problematic studies to participate through professional networks and online forums. Participants were invited to provide feedback on the checks on the list, and were asked to describe any additional checks they knew of, which were not featured in the list. Results Extensive feedback on an initial list of 102 checks was provided by 71 participants based in 16 countries across five continents. Fourteen new checks were proposed across the five domains, and suggestions were made to reword checks on the initial list. An updated list of checks was constructed, comprising 116 checks. Many participants expressed a lack of familiarity with statistical checks, and emphasized the importance of feasibility of the tool. Conclusions A comprehensive list of trustworthiness checks has been produced. The checks will be evaluated to determine which should be included in the INSPECT-SR tool.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

“…The methods used in this study have been described in an online protocol (https://osf.io/6pmx5/) and in a protocol paper describing the INSPECT-SR project (3). We give an overview here.…”

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

A survey of experts to identify methods to detect problematic studies: Stage 1 of the INSPECT-SR Project

Wilkinson,

Heal,

Antoniou

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

The Individual Participant Data Integrity Tool for assessing the integrity of randomised trials

Hunter,

Aberoumand,

Libesman

et al. 2024

Research Synthesis Methods

View full text Add to dashboard Cite

Increasing concerns about the trustworthiness of research have prompted calls to scrutinise studies' Individual Participant Data (IPD), but guidance on how to do this was lacking. To address this, we developed the IPD Integrity Tool to screen randomised controlled trials (RCTs) for integrity issues. Development of the tool involved a literature review, consultation with an expert advisory group, piloting on two IPD meta‐analyses (including 73 trials with IPD), preliminary validation on 13 datasets with and without known integrity issues, and evaluation to inform iterative refinements. The IPD Integrity Tool comprises 31 items (13 study‐level, 18 IPD‐specific). IPD‐specific items are automated where possible, and are grouped into eight domains, including unusual data patterns, baseline characteristics, correlations, date violations, patterns of allocation, internal and external inconsistencies, and plausibility of data. Users rate each item as having either no issues, some/minor issue(s), or many/major issue(s) according to decision rules, and justification for each rating is recorded. Overall, the tool guides decision‐making by determining whether a trial has no concerns, some concerns requiring further information, or major concerns warranting exclusion from evidence synthesis or publication. In our preliminary validation checks, the tool accurately identified all five studies with known integrity issues. The IPD Integrity Tool enables users to assess the integrity of RCTs via examination of IPD. The tool may be applied by evidence synthesists, editors and others to determine whether an RCT should be considered sufficiently trustworthy to contribute to the evidence base that informs policy and practice.

show abstract

Development of the Individual Participant Data (IPD) Integrity Tool for assessing the integrity of randomised trials using individual participant data

Hunter,

Aberoumand,

Libesman

et al. 2023

Preprint

View full text Add to dashboard Cite

IntroductionIncreasing concerns about integrity in medical research have prompted the development of tools to detect untrustworthy studies. Existing tools focus on evaluating aggregate or published data, though some trustworthiness issues may only be detected upon scrutiny of individual participant data (IPD). To address this, we developed the IPD Integrity Tool for detecting integrity issues in randomised controlled trials with IPD available. This manuscript describes the development of this tool.MethodsWe conducted a literature review to collate and map existing integrity items. These were discussed with an expert advisory group, and agreed items were included in a standardised tool and automated where possible. We piloted this tool in two IPD meta-analyses, and conducted preliminary validation checks on 13 datasets with and without known integrity issues in a blinded manner.ResultsThe literature review identified 120 integrity items: 54 could be conducted at the publication or aggregate data (AD) level, 48 required IPD, and 18 were possible with aggregate data, but more comprehensive with IPD. Based on these items, an initial reduced tool was developed in a consensus process involving 13 advisors with different backgrounds (countries, profession, education). This initial tool included 11 items across four domains for AD, and 12 items across 8 domains requiring IPD. The tool was iteratively refined throughout piloting on two IPD meta-analyses including a total of 116 trials (73 with IPD, and 43 with only AD available), and preliminary validation using an additional 13 datasets. All five studies with known integrity issues were accurately identified during validation. The final version of the tool included seven domains with 13 items for AD and eight domains with 18 items requiring IPD.ConclusionsThe quality of evidence informing health care relies on trustworthy data. This manuscript describes the development of a tool to enable researchers, editors, and other stakeholders to detect integrity issues in randomised trials using IPD. Detailed instructions on the application of this tool will be published subsequently.

show abstract

Protocol for the development of a tool (INSPECT-SR) to identify problematic randomised controlled trials in systematic reviews of health interventions

Cited by 4 publications

References 19 publications

A survey of experts to identify methods to detect problematic studies: Stage 1 of the INSPECT-SR Project

A survey of experts to identify methods to detect problematic studies: Stage 1 of the INSPECT-SR Project

The Individual Participant Data Integrity Tool for assessing the integrity of randomised trials

Development of the Individual Participant Data (IPD) Integrity Tool for assessing the integrity of randomised trials using individual participant data

Contact Info

Product

Resources

About