Evaluating the Impact of Database Heterogeneity on Observational Study Results

Abstract

Clinical studies that use observational databases to evaluate the effects of medical products have become commonplace. Such studies begin by selecting a particular database, a decision that published papers invariably report but do not discuss. Studies of the same issue in different databases, however, can and do generate different results, sometimes with strikingly different clinical implications. In this paper, we systematically study heterogeneity among databases, holding other study methods constant, by exploring relative risk estimates for 53 drugoutcome pairs and 2 widely used study designs (cohort studies and self-controlled case series) across 10 observational databases. When holding the study design constant, our analysis shows that estimated relative risks range from a statistically significant decreased risk to a statistically significant increased risk in 11 of 53 (21%) of drug-outcome pairs that use a cohort design and 19 of 53 (36%) of drug-outcome pairs that use a self-controlled case series design. This exceeds the proportion of pairs that were consistent across databases in both direction and statistical significance, which was 9 of 53 (17%) for cohort studies and 5 of 53 (9%) for self-controlled case series. Our findings show that clinical studies that use observational databases can be sensitive to the choice of database. More attention is needed to consider how the choice of data source may be affecting results. database; heterogeneity; methods; population characteristics; reproducibility of results; surveillanceThe increasing use of large-scale observational clinical databases underlies the recent rapid growth in the number of epidemiologic database studies. Such studies seek to use administrative claims data or electronic health records to address important questions about the effects of medical products by using observational study designs. Many potential biases and sources of variability threaten the validity of such studies, and a substantial literature documents these concerns (1-3). Although published studies typically discuss various limitations, such studies rarely discuss alternative databases that could have been used and how the choice of database might have affected results. Indeed, most reports simply identify the data source and provide no discussion about the process used to select it. The literature provides several important examples wherein different studies that used different data sources arrived at contradictory conclusions (4-7). Recent meta-analyses of observational studies have shown that individual studies of the same drug effect yielded conflicting results ranging from statistically significant decreased risk to statistically significant increased risk. Specific examples of meta-analyses wherein individual studies provide conflicting results include a meta-analysis (8) that considered the association between oral contraceptives and endometriosis, a meta-analysis (9) of the effects of proton pump inhibitors on mortality in patients receiving clopidogrel, and a meta-a...

Good Practices for Real‐World Data Studies of Treatment and/or Comparative Effectiveness: Recommendations from the Joint ISPOR‐ISPE Special Task Force on Real‐World Evidence in Health Care Decision Making

“…Confirmation in a second, different database does not guarantee validity. 42,43 There are situations when replication in another data source is for practical reasons impossible. For example, if an association with a rare outcome was detected in the FDA Sentinel data infrastructure of >160 million lives, one would be hard pressed to find an equally sized highquality data source more representative of the at-risk population.…”

Surgical research using national databases

“…Ann Transl Med 2016;4(20):393 atm.amegroups.com that database study results can change from statistical significance in one direction to the opposite direction depending on the database utilized (46).…”

Electronic Health Data for Postmarket Surveillance: A Vision Not Realized

“…An optimal set of parameters for one adverse event turned out to be suboptimal for other adverse events. Also, a higher ranked method, optimized, produced different and sometimes contradictory results among the five different databases for the same adverse event [13]. Finally, some methods revealed substantial evidence of systematic bias, finding statistically significant associations between drugs and events among the negative controls where no relationship was thought to exist.…”

“…Madigan and colleagues [13] provided a graphic portrait of the problem as part of OMOP. The investigators took two widely used analytic methods (new user cohort and selfcontrolled case series) and evaluated the heterogeneity across the six large health claims databases in the project.…”