Improving reproducibility by using high-throughput observational studies with empirical calibration

Abstract

Concerns over reproducibility in science extend to research using existing healthcare data; many observational studies investigating the same topic produce conflicting results, even when using the same data. To address this problem, we propose a paradigm shift. The current paradigm centres on generating one estimate at a time using a unique study design with unknown reliability and publishing (or not) one estimate at a time. The new paradigm advocates for high-throughput observational studies using consistent and standardized methods, allowing evaluation, calibration and unbiased dissemination to generate a more reliable and complete evidence base. We demonstrate this new paradigm by comparing all depression treatments for a set of outcomes, producing 17 718 hazard ratios, each using methodology on par with current best practice. We furthermore include control hypotheses to evaluate and calibrate our evidence generation process. Results show good transitivity and consistency between databases, and agree with four out of the five findings from clinical trials. The distribution of effect size estimates reported in the literature reveals an absence of small or null effects, with a sharp cut-off at p = 0.05. No such phenomena were observed in our results, suggesting more complete and more reliable evidence.This article is part of a discussion meeting issue ‘The growing ubiquity of algorithms in society: implications, impacts and innovations’.

Comprehensive comparative effectiveness and safety of first-line antihypertensive drug classes: a systematic, multinational, large-scale analysis

“…Finally, LEGEND-HTN delivers true Open Science , with all study artifacts including study protocol, analytical code, and full results made publicly available. As a consequence, LEGEND-HTN evidence should demonstrate high reliability 24 . Table 1: Population size and follow-up time for each first-line antihypertensive drug class within each database.…”

Examination of CIs in health and medical journals from 1976 to 2019: an observational study

“…We used a second set of published estimates from health research based on thousands of analyses with no p-hacking or publication bias 13. Using four large databases of insurance claims, the study examined all possible pairs of 17 treatments for depression using 22 outcomes.…”

“…‘Positive’ associations were created by simulating an increased risk with a hazard ratio of 1.5. Details are in the paper by Schuemie et al ;13 the key point for our analysis is that we have a large sample of ratio confidence intervals from Treatment–Outcome pairs that are not subject to any ‘significance seeking’ by researchers or journals.…”

Technology Feature | When robots sleep, do they dream of algorithms?

“…Each of the results meets current methodological standards for publication as a paper in a top peer-reviewed journal. However, the team saw a reassuring distribution of both positive and negative results, indicating they'd avoided the usual publication bias (2).…”