When no answer is better than a wrong answer: a causal perspective on batch effects

Abstract

Batch effects, undesirable sources of variance across multiple experiments, present a substantial hurdle for scientific and clinical discoveries. Specifically, the presence of batch effects can create both spurious discoveries and hide veridical signals, contributing to the ongoing reproducibility crisis. Typical approaches to dealing with batch effects conceptualize 'batches' as an associational effect, rather than a causal effect, despite the fact that the sources of variance that comprise the batch -- potentially including experimental design and population demographics -- causally impact downstream inferences. We therefore cast batch effects as a causal problem rather than an associational problem. This reformulation enables us to make explicit the assumptions and limitations of existing approaches for dealing with batch effects. We therefore develop causal batch effect strategies---Causal Dcorr for discovery of batch effects and Causal ComBat for mitigating batch effects -- which build upon existing statistical associational methods by incorporating modern causal inference techniques. We apply these strategies to a large mega-study of human connectomes assembled by the Consortium for Reliability and Reproducibility, consisting of 24 batches including over 1700 individuals to illustrate that existing approaches create more spurious discoveries (false positives) and miss more veridical signals (true positives) than our proposed approaches. Our work therefore introduces a conceptual framing, as well as open source code, for combining multiple distinct datasets to increase confidence in claims of scientific and clinical discoveries.

ComBatLS: A location- and scale-preserving method for multi-site image harmonization

“…Finally, while popular, the existing cross-sectional ComBat methods assessed here only represent one flavor of multi-site harmonization 4,5,36,37 . It will be important for future work to compare ComBatLS to -or potentially integrate it with -other innovative and emerging frameworks [38][39][40] .…”

Accounting for motion in resting-state fMRI: What part of the spectrum are we characterizing in autism spectrum disorder?

“…We use DRTMLE to estimate the deconfounded group difference. However, a host of other statistical methods could be applied to the same end including covariate matching, propensity score matching ( Bridgeford et al, 2021 ; Stuart, 2010 ), inverse propensity weighting ( Lewinn et al, 2017 ), G-computation ( Robins, 1986 ; Snowden et al, 2011 ), augmented inverse propensity weighting ( Robins et al, 2012 ), and targeted maximum likelihood estimation ( Schuler and Rose, 2017 ; van der Laan and Rubin, 2006 ). Comparing the performance of these approaches in the context of rs-fMRI studies is an important area for future work but is beyond the scope of this paper.…”

Accounting for motion in resting-state fMRI: What part of the spectrum are we characterizing in autism spectrum disorder?

“…We use DRTMLE to estimate the deconfounded group difference. However, a host of other statistical methods could be applied to the same end including covariate matching, propensity score matching (Stuart, 2010; Bridgeford et al, 2021), inverse propensity weighting (Lewinn et al, 2017), G-computation (Robins, 1986; Snowden et al, 2011), augmented inverse propensity weighting (Robins et al, 2012), and targeted maximum likelihood estimation (van der Laan and Rubin, 2006; Schuler and Rose, 2017). Comparing the performance of these approaches in the context of rs-fMRI studies is an important area for future work but is beyond the scope of this paper.…”