A plea to stop using the case‐control design in retrospective database studies

Abstract

The case‐control design is widely used in retrospective database studies, often leading to spectacular findings. However, results of these studies often cannot be replicated, and the advantage of this design over others is questionable. To demonstrate the shortcomings of applications of this design, we replicate two published case‐control studies. The first investigates isotretinoin and ulcerative colitis using a simple case‐control design. The second focuses on dipeptidyl peptidase‐4 inhibitors and acute pancreatitis, using a nested case‐control design. We include large sets of negative control exposures (where the true odds ratio is believed to be 1) in both studies. Both replication studies produce effect size estimates consistent with the original studies, but also generate estimates for the negative control exposures showing substantial residual bias. In contrast, applying a self‐controlled design to answer the same questions using the same data reveals far less bias. Although the case‐control design in general is not at fault, its application in retrospective database studies, where all exposure and covariate data for the entire cohort are available, is unnecessary, as other alternatives such as cohort and self‐controlled designs are available. Moreover, by focusing on cases and controls it opens the door to inappropriate comparisons between exposure groups, leading to confounding for which the design has few options to adjust for. We argue that this design should no longer be used in these types of data. At the very least, negative control exposures should be used to prove that the concerns raised here do not apply.

Considerations for outcome‐dependent biased sampling in health databases

“…I posit that the results of matched and unmatched nested case‐control samples compared to the full health database using the same target parameter are also of key interest. This is well aligned with the settings of retrospective health databases, as noted by Scheumie et al, where all of the exposure and covariate information is available in the full database and “the analysis is performed solely within the resident data.”…”

“…Here, I highlight another major design factor that could cause additional bias lurking in both of the case‐control studies the authors replicated: matching. The concepts I present are by no means new to the literature, but I aim for this commentary to be a useful complement to the work of Scheumie et al…”

“…Odds ratios in three different sample types were considered, ie, the health database, sex‐matched case‐control samples drawn from the health database, and unmatched case‐control samples drawn from health database. Scheumie et al compared odds ratios in matched case‐control samples to incidence rate ratios in self‐controlled case series designs. I posit that the results of matched and unmatched nested case‐control samples compared to the full health database using the same target parameter are also of key interest.…”

“…Scheumie et al issued a plea for researchers to reconsider the usage of case‐control designs constructed in retrospective databases due to concerns related to bias leading to a lack of replicability. Here, I highlight another major design factor that could cause additional bias lurking in both of the case‐control studies the authors replicated: matching.…”

A comment on “a plea to stop using the case‐control design in retrospective database studies”

“…Schuemie et al demonstrate how the case‐control design is vulnerable to between‐person and within‐person confounding, by replicating two case‐control studies in healthcare databases: a study of isotretinoin on the risk of ulcerative colitis in the PharMetrics Patient‐Centric Database and a study of dipeptidyl peptidase‐4 (DPP‐4) inhibitors on the risk of acute pancreatitis in the Taiwan's National Health Insurance Research Database . In summary, they consider that the case‐control study is unnecessary in database studies, where all exposure and covariate data are available for the entire cohort, arguing a lack of reproducibility in such designs.…”