Cautionary tales in the interpretation of observational studies of effects of clinical interventions

Abstract

Observational studies of the effectiveness of clinical interventions are proliferating as more 'real-world' clinical data (so called 'big data') are gathered from clinical registries, administrative datasets and electronic health records. While well-conducted randomised controlled trials (RCT) remain the scientific standard in assessing the efficacy of clinical interventions, well-designed observational studies may add to the evidence base of effectiveness in situations where RCT are of limited value or very difficult to perform. Rather than dismissing observational studies, we need to determine what circumstances may justify doing an observational study and when the study is sufficiently rigorous to be considered reasonably trustworthy. This article proposes criteria by which users of the literature might make such determinations.

INTEREST IN CD2, a global patient-centred study of long-term cervical dystonia treatment with botulinum toxin

“…We report here the baseline characteristics of the INTEREST IN CD2 cohort, which is the largest study in CD to date and was conducted across six continents. While payers have long recognised the need to understand how a clinical population presents for treatment and is managed, clinicians are only starting to understand the relevant insights that real-world studies such as INTEREST IN CD2 can provide [ 15 ]. In particular, the scope of the database allowed us to explore the commonalities and differences in international treatment practices, not only in terms of patient presentation, but also injection practice.…”

Knowledge Discovery and the Aesthetics of Big Data: Simulating the Herschel Observatory

“…. discover connections" [18] or the use of big medical data to find correlations [19], or the habit of natural scientists to call their stunning visual simulations "realistic" (which is just to say that they look right) [20].…”

Hope, hype and harms of Big Data

“…Fourth, analytic tools applied to observational data may be incapable of dealing adequately with confounding, especially confounding by indication, due to the lack of nuanced data on patient and clinician characteristics and behaviour that underpin many healthcare choices . As evidence, patients have often been subjected, in some cases for decades, to ineffective or harmful care (such as hormonal replacement therapy for cardioprotection in post‐menopausal women) based on putative benefits derived from epidemiologic data that did not account for selection bias . Fifth, while Big Data may generate prediction rules that are more discriminatory (more of those predicted to die will in fact die), they may remain poorly calibrated when quantifying risk in individual patients (your predicted risk of death may vary from X% to Y%) .…”

“…19 As evidence, patients have often been subjected, in some cases for decades, to ineffective or harmful care (such as hormonal replacement therapy for cardioprotection in post-menopausal women) based on putative benefits derived from epidemiologic data that did not account for selection bias. 20,21 Fifth, while Big Data may generate prediction rules that are more discriminatory (more of those predicted to die will in fact die), they may remain poorly calibrated when quantifying risk in individual patients (your predicted risk of death may vary from X% to Y%). 22 Sixth, while the proliferation of large datasets of laboratory test results may allow more granular stratification of disease risk, such strata need to be correlated with long term clinical outcomes in defining healthy populations used for determining normal reference ranges.…”