Bias due to selective inclusion and reporting of outcomes and analyses in systematic reviews of randomised trials of healthcare interventions

Abstract

Discrepant outcome reporting between the protocol and published systematic review is fairly common, although the association between statistical significance and discrepant outcome reporting is uncertain. Complete reporting of outcomes in systematic review abstracts is associated with statistical significance of the results for those outcomes. Systematic review outcomes and analysis plans should be specified prior to seeing the results of included studies to minimise post-hoc decisions that may be based on the observed results. Modifications that occur once the review has commenced, along with their justification, should be clearly reported. Effect estimates and CIs should be reported for all systematic review outcomes regardless of the results. The lack of research on selective inclusion of results in systematic reviews needs to be addressed and studies that avoid the methodological weaknesses of existing research are also needed.

PRISMA harms checklist: improving harms reporting in systematic reviews

“…57 There is also evidence that those reporting on primary studies might choose to play down the estimates of harms and emphasise the efficacy of the intervention instead. [56][57][58] Selective outcome reporting and publication bias (where entire studies are unpublished due to the unexpected findings of harms) could therefore work in different directions from that observed with efficacy trials where benefits are emphasised and harms are neglected or distorted. 4 56-63 When statistical approaches are used to probe for the possibility of biased reporting, they should be explicitly described and used with caution.…”

Resuming the discussion of AMSTAR: What can (should) be made better?

“…The intention is to reduce the risk of bias related to selective reporting by stating a priori hypotheses and methods explicitly [18]. However, a thorough evaluation should focus on discrepancies between the protocol and the final review, which research has shown to be fairly common [19, 20]. It is not sufficient only to refer to a protocol, therefore, as this would primarily reflect the quality of reporting.…”

The Journal Impact Factor: Moving Toward an Alternative and Combined Scientometric Approach

“…The ‘obsession’ with articles attracting abundant citations may be also the trigger of the current unprecedented proliferation of systematic reviews (14), most of which are of low quality and even harmful for the scientific evidence accumulation (151617). …”