Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy

Mayo‐Wilson, Evan; Li, Tianjing; Fusco, Nicole; Bertizzolo, Lorenzo; Canner, Joseph K.; Cowley, Terrie; Doshi, Peter; Ehmsen, Jeffrey T.; Gresham, Gillian; Guo, Nan; Haythornthwaite, Jennifer A.; Heyward, James; Hong, Hwanhee; Pham, Diana; Payne, Jennifer L.; Rosman, Lori; Stuart, Elizabeth A.; Suarez‐Cuervo, Catalina; Tolbert, Elizabeth; Twose, Claire; Vedula, S. Swaroop; Dickersin, Kay

doi:10.1016/j.jclinepi.2017.07.014

Cited by 90 publications

(91 citation statements)

References 55 publications

(62 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When multiple sources are used, multiple syntheses containing many outcomes may be possible . In the MUDS study, as in previous studies, it was possible to change the results of meta‐analyses by selectively choosing information from multiple data sources.…”

Section: Step 4 Prespecify Outcomes and Results To Examine In The Symentioning

confidence: 99%

“…Reanalyzing IPD requires more resources and expertise compared with reviewing aggregated data, including time from statisticians and senior investigators. In the MUDS study, gabapentin IPD were in electronic databases, and quetiapine IPD were in tables as appendices to a CSR, and we converted the data to an electronic format; IPD were not accompanied by metadata in either case . Furthermore, some systems used to record data in older trials are not in current use.…”

Section: Step 1 Specify Which Sources You Will Use For Your Review Amentioning

confidence: 99%

“…However, some sources do not include identifiers, and other sources include incorrect identifiers, so methods for linking sources are difficult to automate . Sometimes multiple sources about the same trial do not reference each other, do not share common authors, do not include enough information about their design and results to determine whether they describe the same trial, or include different information about the same trial . In addition, some sources are undated; in these cases, it may be difficult to determine whether the sources are related to the same trial and which data are most current.…”

Section: Step 2 Link Each Individual Trial With Its Data Sourcesmentioning

confidence: 99%

“…We found that trials included many outcomes and results, which created opportunities for selective reporting by both trialists and systematic reviewers . By including information from multiple data sources selectively, we found that a systematic reviewer could manipulate the results and interpretation of individual trials and meta‐analyses …”

Section: Introductionmentioning

confidence: 99%

“…2 By including information from multiple data sources selectively, we found that a systematic reviewer could manipulate the results and interpretation of individual trials and metaanalyses. 3 In this guidance, we focus on existing data sources (eg, reports and databases). Systematic reviewers sometimes contact trialists to ask questions about study design or to request specific results, and these issues have been considered elsewhere.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Practical guidance for using multiple data sources in systematic reviews and meta‐analyses (with examples from the MUDS study)

Mayo‐Wilson

Fusco

et al. 2017

Research Synthesis Methods

Self Cite

View full text Add to dashboard Cite

Data for individual trials included in systematic reviews may be available in multiple sources. For example, a single trial might be reported in 2 journal articles and 3 conference abstracts. Because of differences across sources, source selection can influence the results of systematic reviews. We used our experience in the Multiple Data Sources in Systematic Reviews (MUDS) study, and evidence from previous studies, to develop practical guidance for using multiple data sources in systematic reviews.We recommend the following: (1) Specify which sources you will use. Before beginning a systematic review, consider which sources are likely to contain the most useful data. Try to identify all relevant reports and to extract information from the most reliable sources. (2) Link individual trials with multiple sources. Write to authors to determine which sources are likely related to the same trials. Use a modified Preferred Reporting Items for Systematic Reviews and Meta‐analyses (PRISMA) flowchart to document both the selection of trials and the selection of sources. (3) Follow a prespecified protocol for extracting trial characteristics from multiple sources. Identify differences among sources, and contact study authors to resolve differences if possible. (4) Prespecify outcomes and results to examine in the review and meta‐analysis. In your protocol, describe how you will handle multiple outcomes within each domain of interest. Look for outcomes using all eligible sources. (5) Identify which data sources were included in the review. Consider whether the results might have been influenced by data sources used. (6) To reduce bias, and to reduce research waste, share the data used in your review.

show abstract

Section: Step 4 Prespecify Outcomes and Results To Examine In The Symentioning

confidence: 99%

Section: Step 1 Specify Which Sources You Will Use For Your Review Amentioning

confidence: 99%

Section: Step 2 Link Each Individual Trial With Its Data Sourcesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Practical guidance for using multiple data sources in systematic reviews and meta‐analyses (with examples from the MUDS study)

Mayo‐Wilson

Fusco

et al. 2017

Research Synthesis Methods

Self Cite

View full text Add to dashboard Cite

show abstract

Guidelines for Reporting Outcomes in Trial Reports

Butcher

Monsour

Mew

et al. 2022

JAMA

Self Cite

194

105

View full text Add to dashboard Cite

ImportanceClinicians, patients, and policy makers rely on published results from clinical trials to help make evidence-informed decisions. To critically evaluate and use trial results, readers require complete and transparent information regarding what was planned, done, and found. Specific and harmonized guidance as to what outcome-specific information should be reported in publications of clinical trials is needed to reduce deficient reporting practices that obscure issues with outcome selection, assessment, and analysis.ObjectiveTo develop harmonized, evidence- and consensus-based standards for reporting outcomes in clinical trial reports through integration with the Consolidated Standards of Reporting Trials (CONSORT) 2010 statement.Evidence ReviewUsing the Enhancing the Quality and Transparency of Health Research (EQUATOR) methodological framework, the CONSORT-Outcomes 2022 extension of the CONSORT 2010 statement was developed by (1) generation and evaluation of candidate outcome reporting items via consultation with experts and a scoping review of existing guidance for reporting trial outcomes (published within the 10 years prior to March 19, 2018) identified through expert solicitation, electronic database searches of MEDLINE and the Cochrane Methodology Register, gray literature searches, and reference list searches; (2) a 3-round international Delphi voting process (November 2018-February 2019) completed by 124 panelists from 22 countries to rate and identify additional items; and (3) an in-person consensus meeting (April 9-10, 2019) attended by 25 panelists to identify essential items for the reporting of outcomes in clinical trial reports.FindingsThe scoping review and consultation with experts identified 128 recommendations relevant to reporting outcomes in trial reports, the majority (83%) of which were not included in the CONSORT 2010 statement. All recommendations were consolidated into 64 items for Delphi voting; after the Delphi survey process, 30 items met criteria for further evaluation at the consensus meeting and possible inclusion in the CONSORT-Outcomes 2022 extension. The discussions during and after the consensus meeting yielded 17 items that elaborate on the CONSORT 2010 statement checklist items and are related to completely defining and justifying the trial outcomes, including how and when they were assessed (CONSORT 2010 statement checklist item 6a), defining and justifying the target difference between treatment groups during sample size calculations (CONSORT 2010 statement checklist item 7a), describing the statistical methods used to compare groups for the primary and secondary outcomes (CONSORT 2010 statement checklist item 12a), and describing the prespecified analyses and any outcome analyses not prespecified (CONSORT 2010 statement checklist item 18).Conclusions and RelevanceThis CONSORT-Outcomes 2022 extension of the CONSORT 2010 statement provides 17 outcome-specific items that should be addressed in all published clinical trial reports and may help increase trial utility, replicability, and transparency and may minimize the risk of selective nonreporting of trial results.

show abstract

Guidelines for Reporting Outcomes in Trial Protocols

Butcher

Monsour

Mew

et al. 2022

JAMA

Self Cite

View full text Add to dashboard Cite

ImportanceComplete information in a trial protocol regarding study outcomes is crucial for obtaining regulatory approvals, ensuring standardized trial conduct, reducing research waste, and providing transparency of methods to facilitate trial replication, critical appraisal, accurate reporting and interpretation of trial results, and knowledge synthesis. However, recommendations on what outcome-specific information should be included are diverse and inconsistent. To improve reporting practices promoting transparent and reproducible outcome selection, assessment, and analysis, a need for specific and harmonized guidance as to what outcome-specific information should be addressed in clinical trial protocols exists.ObjectiveTo develop harmonized, evidence- and consensus-based standards for describing outcomes in clinical trial protocols through integration with the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) 2013 statement.Evidence ReviewUsing the Enhancing the Quality and Transparency of Health Research (EQUATOR) methodological framework, the SPIRIT-Outcomes 2022 extension of the SPIRIT 2013 statement was developed by (1) generation and evaluation of candidate outcome reporting items via consultation with experts and a scoping review of existing guidance for reporting trial outcomes (published within the 10 years prior to March 19, 2018) identified through expert solicitation, electronic database searches of MEDLINE and the Cochrane Methodology Register, gray literature searches, and reference list searches; (2) a 3-round international Delphi voting process (November 2018-February 2019) completed by 124 panelists from 22 countries to rate and identify additional items; and (3) an in-person consensus meeting (April 9-10, 2019) attended by 25 panelists to identify essential items for outcome-specific reporting to be addressed in clinical trial protocols.FindingsThe scoping review and consultation with experts identified 108 recommendations relevant to outcome-specific reporting to be addressed in trial protocols, the majority (72%) of which were not included in the SPIRIT 2013 statement. All recommendations were consolidated into 56 items for Delphi voting; after the Delphi survey process, 19 items met criteria for further evaluation at the consensus meeting and possible inclusion in the SPIRIT-Outcomes 2022 extension. The discussions during and after the consensus meeting yielded 9 items that elaborate on the SPIRIT 2013 statement checklist items and are related to completely defining and justifying the choice of primary, secondary, and other outcomes (SPIRIT 2013 statement checklist item 12) prospectively in the trial protocol, defining and justifying the target difference between treatment groups for the primary outcome used in the sample size calculations (SPIRIT 2013 statement checklist item 14), describing the responsiveness of the study instruments used to assess the outcome and providing details on the outcome assessors (SPIRIT 2013 statement checklist item 18a), and describing any planned methods to account for multiplicity relating to the analyses or interpretation of the results (SPIRIT 2013 statement checklist item 20a).Conclusions and RelevanceThis SPIRIT-Outcomes 2022 extension of the SPIRIT 2013 statement provides 9 outcome-specific items that should be addressed in all trial protocols and may help increase trial utility, replicability, and transparency and may minimize the risk of selective nonreporting of trial results.

show abstract

Cherry-picking by trialists and meta-analysts can drive conclusions about intervention efficacy

Abstract: Disagreements across data sources affect the effect size, statistical significance, and interpretation of trials and meta-analyses.

Cited by 90 publications

References 55 publications

Practical guidance for using multiple data sources in systematic reviews and meta‐analyses (with examples from the MUDS study)

Practical guidance for using multiple data sources in systematic reviews and meta‐analyses (with examples from the MUDS study)

Guidelines for Reporting Outcomes in Trial Reports

Guidelines for Reporting Outcomes in Trial Protocols

Contact Info

Product

Resources

About