Using Negative Control Outcomes and Difference-in-Differences Analysis to Estimate Treatment Effects in an Entirely Treated Cohort: The Effect of Ivacaftor in Cystic Fibrosis

Abstract

When an entire cohort of patients receives a treatment it is difficult to estimate the treatment effect in the treated because there are no directly comparable untreated patients. Attempts can be made to find a suitable control group, (e.g. historical controls), but underlying differences between the treated and untreated can result in bias. We show how negative control outcomes (NCO) combined with difference-in-differences analysis can be used to assess bias in treatment effect estimates and obtain unbiased estimates under certain assumptions. Causal diagrams and potential outcomes are used to explain the methods and assumptions. We apply the methods to UK Cystic Fibrosis (CF) Registry data to investigate the effect of ivacaftor, introduced in 2012 for a subset of the CF population with a particular genotype, on lung function and days receiving intravenous antibiotics (IV days). We consider two NCOs: outcomes measured in the pre-ivacaftor period and outcomes in individuals ineligible for ivacaftor due to their genotype. Ivacaftor was found to improve lung function in year one (~6.5 increase in FEV1%), was associated with reduced lung function decline (~0.5 decrease in annual FEV1% decline, though confidence intervals include 0), and reduced the rate of IV days (~60% over 3 years).

The COVID-19 pandemic and energy transitions: Evidence from low-carbon power generation in China

“…All Chinese provinces were in some degree of lockdown during the pandemic period, meaning that observational data at the province level provided no contemporary untreated controls, and it was difficult to estimate an average treatment effect according to the standard DID model. The literature proposed to identify a comparable group that could not receive treatment, e.g., historical controls prior to its availability ( Newsome et al, 2021 ; He et al, 2020 ). With reference to Wang et al (2021) , how the COVID-19 or national-level pandemic-related measures have affected low-carbon generation relative to the trends in previous periods was examined and the first modified DID model with historical controls was as follows.…”

Regarding the article entitled “Effect of elexacaftor/tezacaftor/ivacaftor on annual rate of lung function decline in people with cystic fibrosis”

“…This may bias the analysis by Lee et al [1] in favour of ELX/TEZ/IVA treated participants. Newsome et al [5] have previously pointed out bias in using historical cohorts as control groups to estimate the treatment effect of CFTR modulators. This is particularly problematic if the treatment follow-up coincided with Covid-19 pandemic which may be associated with a substantially improved FEV 1 trend even among people not using ELX/TEZ/IVA, probably from reduced exposure to respiratory viruses and a reduction in pulmonary exacerbations [2].…”

“…This is particularly problematic if the treatment follow-up coincided with Covid-19 pandemic which may be associated with a substantially improved FEV 1 trend even among people not using ELX/TEZ/IVA, probably from reduced exposure to respiratory viruses and a reduction in pulmonary exacerbations [2]. In estimating the treatment effect of ELX/TEZ/IVA, it may be more appropriate to use negative control outcomes combined with difference-in-differences analysis as proposed by Newsome et al [5] instead of comparison against historical cohorts. Longer-term follow-up of ELX/TEZ/IVA treated participants in the post-Covid epoch is also important, as people with CF start returning to their pre-pandemic lifestyle.…”