Variability and Bias in Microbiome Metagenomic Sequencing: an Interlaboratory Study Comparing Experimental Protocols

Abstract

Background: Several studies have documented the significant impact of methodological choices in microbiome analyses. The myriad of methodological options available complicate the replication of results and generally limit the comparability of findings between independent studies that use differing techniques and measurement pipelines. Here we describe the Mosaic Standards Challenge (MSC), an international interlaboratory study designed to assess the impact of methodological variables on the results. The MSC did not prescribe methods but rather asked participating labs to analyze 7 shared reference samples (5x human stool samples and 2x mock communities) using their standard laboratory methods. To capture the array of methodological variables, each participating lab completed a metadata reporting sheet that included 100 different questions regarding the details of their protocol. The goal of this study was to survey the methodological landscape for microbiome metagenomic sequencing (MGS) analyses and the impact of methodological decisions on metagenomic sequencing results. Results: A total of 44 labs participated in the MSC by submitting results (16S or WGS) along with accompanying metadata; thirty 16S rRNA gene amplicon datasets and 14 WGS datasets were collected. The inclusion of two types of reference materials (human stool and mock communities) enabled analysis of both MGS measurement variability between different protocols using the biologically-relevant stool samples, and MGS bias with respect to ground truth values using the DNA mixtures. Owing to the compositional nature of MGS measurements, analyses were conducted on the ratio of Firmicutes: Bacteroidetes allowing us to directly apply common statistical methods. The resulting analysis demonstrated that protocol choices have significant effects, including both bias of the MGS measurement associated with a particular methodological choices, as well as effects on measurement robustness as observed through the spread of results between labs making similar methodological choices. In the analysis of the DNA mock communities, MGS measurement bias was observed even when there was general consensus among the participating laboratories. Conclusion: This study was the result of a collaborative effort that included academic, commercial, and government labs. In addition to highlighting the impact of different methodological decisions on MGS result comparability, this work also provides insights for consideration in future microbiome measurement study design.

Evaluating the Analytical Performance of Direct-to-Consumer Gut Microbiome Testing Services

“…The primary contributor to this outcome is methodological variability. The significant impact of methodological variables on the results of microbiome metagenomic measurements has been demonstrated by us and others, and DTC microbiome testing companies are not immune to these sources of bias [14, 17-20]. The use of a homogenous stool reference material enabled a systematic evaluation without the confounding factors of biological or composition heterogeneity that may be found in a traditional stool sample.…”

“…While these case studies are often intriguing and provide some anecdotal insight, to our knowledge, a rigorous evaluation of the analytical performance has not yet been conducted. The National Institute of Standards and Technology (NIST) has developed a suite of candidate human fecal standards, confirmed to be homogeneous and stable via rigorous multi’omic analyses [14, 15]. These materials have been designed to enable stakeholders to evaluate the impact of methodological variability on their microbiome measurements, both qualitatively and quantitatively.…”

A Sensitivity Analysis of Methodological Variables Associated with Microbiome Measurements

“…Samples used in this study were previously described. (15) Briefly, five samples were generated from five different donors. Each sample was prepared by pooling and homogenizing 4 consecutive bowel movements from the donor, combining homogenized material with Omnigene Gut Solution, spiking with 10 8 CFU/mL of Aliivibrio-fischeri (Gram negative, formerly known as Vibrio fischeri ) and Leifsonia xyli (Gram positive), and then aliquoting 1 mL aliquots.…”

“…We previously described the Mosaic Standards Challenge (MSC), an international interlaboratory study designed to assess the impact of methodological variability on MGS results. (15) The MSC employed five biologically distinct human fecal reference materials and a comprehensive standardized metadata reporting sheet that allowed participants to share exhaustive details about the protocols and methods used for analyzing the fecal materials. The MSC employed an "open protocol" design that allowed (and even encouraged) participants to follow divergent protocols as defined for their lab's routine MGS measurements.…”