Interpretive analysis of 85 systematic reviews suggests that narrative syntheses and meta‐analyses are incommensurate in argumentation

Abstract

Using Toulmin's argumentation theory, we analysed the texts of systematic reviews in the area of workplace health promotion to explore differences in the modes of reasoning embedded in reports of narrative synthesis as compared with reports of meta‐analysis. We used framework synthesis, grounded theory and cross‐case analysis methods to analyse 85 systematic reviews addressing intervention effectiveness in workplace health promotion. Two core categories, or ‘modes of reasoning’, emerged to frame the contrast between narrative synthesis and meta‐analysis: practical–configurational reasoning in narrative synthesis (‘what is going on here? What picture emerges?’) and inferential–predictive reasoning in meta‐analysis (‘does it work, and how well? Will it work again?’). Modes of reasoning examined quality and consistency of the included evidence differently. Meta‐analyses clearly distinguished between warrant and claim, whereas narrative syntheses often presented joint warrant–claims. Narrative syntheses and meta‐analyses represent different modes of reasoning. Systematic reviewers are likely to be addressing research questions in different ways with each method. It is important to consider narrative synthesis in its own right as a method and to develop specific quality criteria and understandings of how it is carried out, not merely as a complement to, or second‐best option for, meta‐analysis. © 2016 The Authors. Research Synthesis Methods published by John Wiley & Sons Ltd.

Lack of transparency in reporting narrative synthesis of quantitative data: a methodological assessment of systematic reviews

“…Increasingly, systematic reviews need to address questions about complex interventions and go beyond straightforward questions of effectiveness [3], [4], [19], [25], [26], [27], [28]. This issue goes beyond public health; the Cochrane 2020 strategy points to a move toward incorporating more diverse sources of evidence and addressing complex health decision-making questions [29].…”

“…NS is well placed to support these types of reviews, not only as an alternative when meta-analysis is contraindicated but also as an important synthesis tool in its own right. It offers a method for exploring and understanding the underlying arguments and justification of claims made in the included studies of a review [28]. NS enables reviewers to incorporate diversity in study designs, participants, interventions, or outcomes.…”

The use of ‘PICO for synthesis’ and methods for synthesis without meta-analysis: protocol for a survey of current practice in systematic reviews of health interventions

“…Importantly, the use of other synthesis methods may alter the nature of the question answered by the review and the type of reasoning used to reach conclusions 2 , 13 .…”

Just how plain are plain tobacco packs: re-analysis of a systematic review using multilevel meta-analysis suggests lessons about the comparative benefits of synthesis methods

“…The debate on the relative merits of narrative synthesis and meta-analysis is a well-rehearsed one in the field of systematic review methods [ 1 ]. Yet to our knowledge, a ‘within-review’ examination of the comparative benefits of each synthesis method has not been undertaken recently.…”

“…Previous work by our group comparing narrative syntheses and meta-analyses has suggested that these two approaches may be answering different questions via different approaches, rather than answering the same questions using the typically preferred method (meta-analysis) or its ‘backup’ (narrative synthesis) [ 1 ]. In this work, we described that narrative syntheses and meta-analyses use different modes of reasoning to answer related but distinct questions; in the case of narrative synthesis, the question most often asked is ‘what is going on here?’ or ‘what picture emerges?’ whereas in meta-analysis, the question most often asked is ‘does it work and how well?’ and ‘will it work again?’.…”