Risk of bias in observational studies using routinely collected data of comparative effectiveness research: a meta-research study

Abstract

Background To assess the completeness of reporting, research transparency practices, and risk of selection and immortal bias in observational studies using routinely collected data for comparative effectiveness research. Method We performed a meta-research study by searching PubMed for comparative effectiveness observational studies evaluating therapeutic interventions using routinely collected data published in high impact factor journals from 01/06/2018 to 30/06/2020. We assessed the reporting of the study design (i.e., eligibility, treatment assignment, and the start of follow-up). The risk of selection bias and immortal time bias was determined by assessing if the time of eligibility, the treatment assignment, and the start of follow-up were synchronized to mimic the randomization following the target trial emulation framework. Result Seventy-seven articles were identified. Most studies evaluated pharmacological treatments (69%) with a median sample size of 24,000 individuals. In total, 20% of articles inadequately reported essential information of the study design. One-third of the articles (n = 25, 33%) raised some concerns because of unclear reporting (n = 6, 8%) or were at high risk of selection bias and/or immortal time bias (n = 19, 25%). Only five articles (25%) described a solution to mitigate these biases. Six articles (31%) discussed these biases in the limitations section. Conclusion Reporting of essential information of study design in observational studies remained suboptimal. Selection bias and immortal time bias were common methodological issues that researchers and physicians should be aware of when interpreting the results of observational studies using routinely collected data.

The Innovative Medicines Fund: a universal model for faster and fairer access to new promising medicines or a Trojan horse for low-value creep?

“…26 Using the components of this framework, a recent study assessed the risk of bias in observational studies evaluating therapeutic interventions, with one-third of studies raising concerns because of unclear reporting or high risk of selection bias (confounding) and immortal time bias. 27 This adds to other evidence showing the overall poor quality of RWD studies appraising the clinical benefits of approved therapies by the European Medicines Agency and US Food and Drug Administration, 28 with only 2% of the 293 RWD studies appraised using population-based registry data. In this regard, the NHS has no excuse for not using the administrative and clinical data at its disposal, including the various national clinical audit programmes.…”

Real-World Use of Symptomatic Treatments in Early Alzheimer’s Disease

“…Limited information was reported on missing data or timing of diagnosis and prescription such that changes in diagnosis or treatment may not be adequately captured. Such findings are common in real-world data, and a recent report [ 56 ] found that a third of all real-world evidence studies had limited reporting reflecting the challenges of meta-research of observational data. The GERAS [ 29, 32 ] and GERAS II [ 31, 36 ] databases featured quite heavily in our review focusing on different countries or timeframes.…”

Data source profile reporting by studies that use routinely collected health data to explore the effects of drug treatment

“…RCD was defined as data that were generated for administrative or clinical purposes without a priori research goals [1,5]. Typical RCD includes electronic medical records (EMR), administrative claims data, safety surveillance databases, and pharmacy data [1,7]. Studies that could not confirm whether the data resources were collected independently of prior research goals or that used at least one actively collected data element for research purposes were excluded.…”

“…These data contain information on drug exposures and outcomes that are essential for pharmacoepidemiology studies [1,5]. The added support of information technologies that enable the storage of large datasets, including details on drug use, clinical management, laboratory test results, and patient outcomes, has led to a proliferation in observational pharmacoepidemiology studies in recent decades [3,6,7].…”