Bias Modelling in Evidence Synthesis

Abstract

Summary. Policy decisions often require synthesis of evidence from multiple sources, and the source studies typically vary in rigour and in relevance to the target question. We present simple methods of allowing for differences in rigour (or lack of internal bias) and relevance (or lack of external bias) in evidence synthesis. The methods are developed in the context of reanalysing a UK National Institute for Clinical Excellence technology appraisal in antenatal care, which includes eight comparative studies. Many were historically controlled, only one was a randomized trial and doses, populations and outcomes varied between studies and differed from the target UK setting. Using elicited opinion, we construct prior distributions to represent the biases in each study and perform a bias-adjusted meta-analysis. Adjustment had the effect of shifting the combined estimate away from the null by approximately 10%, and the variance of the combined estimate was almost tripled. Our generic bias modelling approach allows decisions to be based on all available evidence, with less rigorous or less relevant studies downweighted by using computationally simple methods.

Comparative clinical effectiveness of management strategies for sciatica: systematic review and network meta-analyses

“…There is also scope to develop more sophisticated methods, such as building on the confidence profile method, 170 bias-adjusted results, 174 or Bayesian statistics, 169 to incorporate information relating to differences in study design or internal and external validity in the network meta-analyses, as well as data on multiple follow-up periods. The issue of how best to estimate the effectiveness of treatment approaches according to their order within a sequential treatment pathway remains an important challenge.…”

“…Non-randomised studies were included in the search because some treatment approaches may not have been evaluated by RCTs, and also to increase the precision of the findings for interventions evaluated by a limited number of studies. Observational studies can have better external validity than RCTs 169,170 and provide more generalisable findings. However, observational studies are likely to be affected by selection bias and confounding, and may therefore yield estimates of association that deviate from the true underlying relationship beyond the play of chance.…”

The assessment and appraisal of regenerative medicines and cell therapy products: an exploration of methods for review, economic evaluation and appraisal

“…Turner et al, 53 recognising the practical limitations of basing adjustment on external empirical data, proposed an alternative approach in which the direction and magnitude of biases are elicited by reviewers. This method can deal with multiple sources and types of bias including both internal validity bias and external validity bias.…”

“…This method can deal with multiple sources and types of bias including both internal validity bias and external validity bias. In brief, Turner et al 53 proposed that authors design an idealised study aimed at answering the specific question in mind. This study may not be feasible to carry out and is simply a tool for exploring bias in the completed studies.…”

“…[44][45][46][47][48][49][50][51][52][53] These articles included two comprehensive reviews 44,45 as well as individual articles, all of which were identified in the review articles. Many of the techniques described in Verde and Ohmann 44 and Doi, 45 however, have limited applicability to regenerative medicine (i.e.…”