Leveraging vibration of effects analysis for robust discovery in observational biomedical data science

Abstract

Hypothesis generation in observational, biomedical data science often starts with computing an association or identifying the statistical relationship between a dependent and an independent variable. However, the outcome of this process depends fundamentally on modeling strategy, with differing strategies generating what can be called “vibration of effects” (VoE). VoE is defined by variation in associations that often lead to contradictory results. Here, we present a computational tool capable of modeling VoE in biomedical data by fitting millions of different models and comparing their output. We execute a VoE analysis on a series of widely reported associations (e.g., carrot intake and eyesight) with an extended additional focus on lifestyle exposures (e.g., physical activity) and components of the Framingham Risk Score for cardiovascular health (e.g., blood pressure). We leveraged our tool for potential confounder identification, investigating what adjusting variables are responsible for conflicting models. We propose modeling VoE as a critical step in navigating discovery in observational data, discerning robust associations, and cataloging adjusting variables that impact model output.

Systematically assessing microbiome–disease associations identifies drivers of inconsistency in metagenomic research

“…These analyses may be particularly useful for discovery-based studies (very common in the microbiome and genomic fields), approaches designed to generate, rather than test specific candidate, hypotheses from complex datasets. We refer to the distribution of possible associations that emerge from different modeling scenarios when carrying out discovery as vibration of effects (VoE) [ 3 , 13 – 15 ]. In some cases, slight changes to model specification yield polar opposite results (e.g., a particular microbiome feature being both negatively and positively associated with disease) [ 16 ].…”

A realistic benchmark for the identification of differentially abundant taxa in (confounded) human microbiome studies

“…Similar to LEfSe, metadeconfoundR 35,47,48 employs non-parametric tests to first screen for naive associations, but goes a step further to construct multiple mixed-effect models (from the lme4 package 49 ) and apply iterative nested model testing procedures to further classify feature robustness or susceptibility to confounding. Analogous logic may be found in the vibration-of-effects paradigm 50,51 . In our analysis of drug confounding in T2D, we demonstrated the importance of integrating information across covariate-aware association models to reveal robust disease-associated microbial features.…”

“…These samples were saved via their indices such that each method was applied to the exact same data. Seven different sample sizes were explored (12,24,50,100,200, 400, and 800) and 50 sets of test indices were created for each. For the evaluation of a single DA method, a total of 980,000 unique configurations were generated and used as input (7 abundance shifts x 4 prevalence shifts x 100 simulation repeats x 7 sample sizes x 50 repeats).…”

Dynamic effects of bilingualism on brain structure map onto general principles of experience-based neuroplasticity

“…We followed a two-step procedure employed in previous studies using mixed models [60][61][62] . In a first-level model, we fitted the regression splines for the main effect of bilingual experiences using BCS along with the main effect of Age, and Participant and Gender as random effects.…”