Interpreting systematic reviews: are we ready to make our own conclusions? A cross-sectional study

Abstract

BackgroundIndependent evaluation of clinical evidence is advocated in evidence-based medicine (EBM). However, authors' conclusions are often appealing for readers who look for quick messages. We assessed how well a group of Malaysian hospital practitioners and medical students derived their own conclusions from systematic reviews (SRs) and to what extent these were influenced by their prior beliefs and the direction of the study results.MethodsWe conducted two cross-sectional studies: one with hospital practitioners (n = 150) attending an EBM course in June 2008 in a tertiary hospital and one with final-year medical students (n = 35) in November 2008. We showed our participants four Cochrane SR abstracts without the authors' conclusions. For each article, the participants chose a conclusion from among six options comprising different combinations of the direction of effect and the strength of the evidence. We predetermined the single option that best reflected the actual authors' conclusions and labelled this as our best conclusion. We compared the participants' choices with our predetermined best conclusions. Two chosen reviews demonstrated that the intervention was beneficial ("positive"), and two others did not ("negative"). We also asked the participants their prior beliefs about the intervention.ResultsOverall, 60.3% correctly identified the direction of effect, and 30.1% chose the best conclusions, having identified both the direction of effect and the strength of evidence. More students (48.2%) than practitioners (22.2%) chose the best conclusions (P < 0.001). Fewer than one-half (47%) correctly identified the direction of effect against their prior beliefs. "Positive" SRs were more likely than "negative" SRs to change the participants' beliefs about the effect of the intervention (relative risk (RR) 1.8, 95% confidence interval 1.3 to 2.6) and "convert" those who were previously unsure by making them choose the appropriate direction of effect (RR 1.9, 95% confidence interval 1.3 to 2.8).ConclusionsThe majority of our participants could not generate appropriate conclusions from SRs independently. Judicious direction from the authors' conclusions still appears crucial to guiding our health care practitioners in identifying appropriate messages from research. Authors, editors and reviewers should ensure that the conclusions of a paper accurately reflect the results. Similar studies should be conducted in other settings where awareness and application of EBM are different.Please see Commentary: http://www.biomedcentral.com/1741-7015/9/31/.

Evaluation of AMSTAR to assess the methodological quality of systematic reviews in overviews of reviews of healthcare interventions

“…Results and conclusions from each SR were classified based on published criteria [33, 34]. Results were classified as “favourable” ( p  < 0.10 in favour of the intervention, or finding described as ‘significant’), “neutral” ( p  > 0.10, or finding described as ‘not different between groups’), or “unfavourable” ( p  < 0.10 in favour of the comparator, or finding described as ‘favouring non-intervention comparator’).…”

Reporting completeness of abstracts of systematic reviews published in leading dental specialty journals

“…Despite increasing availability of free access and subscription, limited access to full‐text articles and journal subscription costs may hinder retrieval of the complete article. This is particularly problematic as a range of factors, including authors' conclusions, the preconceptions of the reader, and positive findings in SR abstracts, are known to bias readers' interpretation and hence to influence clinical decisions . To encourage transparency and consistency in the reporting of abstracts the CONSORT statement for abstracts of RCTs has been developed .…”

Side effects are incompletely reported among systematic reviews in gastroenterology

“…In the discussion sections, we documented a widespread tendency to misinterpret harmsrelated data, with reported adverse events not acknowledged, and rarely discussed as part of the balance of benefits and harms of an intervention. The impact of misrepresentation may be compounded if studies are accepted as written without further critical appraisal [45,46]. To reduce the impact, and in light of our findings, we suggest that the use of reporting guidelines such as PRISMA be recommended by editors, in line with other recent opinion [47].…”