A fully scalable online pre-processing algorithm for short oligonucleotide microarray atlases

Abstract

Rapid accumulation of large and standardized microarray data collections is opening up novel opportunities for holistic characterization of genome function. The limited scalability of current preprocessing techniques has, however, formed a bottleneck for full utilization of these data resources. Although short oligonucleotide arrays constitute a major source of genome-wide profiling data, scalable probe-level techniques have been available only for few platforms based on pre-calculated probe effects from restricted reference training sets. To overcome these key limitations, we introduce a fully scalable online-learning algorithm for probe-level analysis and pre-processing of large microarray atlases involving tens of thousands of arrays. In contrast to the alternatives, our algorithm scales up linearly with respect to sample size and is applicable to all short oligonucleotide platforms. The model can use the most comprehensive data collections available to date to pinpoint individual probes affected by noise and biases, providing tools to guide array design and quality control. This is the only available algorithm that can learn probe-level parameters based on sequential hyperparameter updates at small consecutive batches of data, thus circumventing the extensive memory requirements of the standard approaches and opening up novel opportunities to take full advantage of contemporary microarray collections.

Effects of bowel cleansing on the intestinal microbiota

“…It targets two hypervariable regions (V1 and V6) of the 16S rRNA gene and covers over 1000 bacterial phylotypes detected in the human GI track. The raw signal intensities were normalised as described previously using by Robust Probabilistic Averaging method8 19 that takes into account the possible cross-hybridisation of the probes. The probe signal intensities were summarised into 130 genus-like taxonomic groups referred to with a species name and relatives according to the nearest cultured relative, for example, Escherichia coli et rel .…”

Comparison of the effects of five dietary fibers on mucosal transcriptional profiles, and luminal microbiota composition and SCFA concentrations in murine colon

“…Quantification of the overall bacterial community density was performed by qPCR targeting the 16 rRNA gene, whereas the microbial community composition was analyzed using the Mouse Intestinal Tract Chip (MITChip) (for further details also see Supporting Information Methods). The relative abundance of 96 genus‐level bacterial groups detected on the MITchip was determined by the Robust Probabilistic Averaging algorithm . To assess the correlation of the microbial groups with all diets groups, multivariate redundancy analysis was performed as implemented in Canoco for Windows 4.5 .…”

Fat, fibre and cancer risk in African Americans and rural Africans

“…Microbiota profiles were generated by pre-processing the probe-level measurements with min-max normalization and the frozen-RPA probe summarization 62 , 63 into three phylogenetic levels: level 1, defined as order-like 16S rRNA gene sequence groups; level 2, defined as genus-like 16S rRNA gene sequence groups (sequence similarity >90%); and level 3, phylotype-like 16S rRNA gene sequence groups (sequence similarity >98%). In the present work we primarily focus on the genus-level (level 2) variation.…”