Sparsification of large ultrametric matrices: insights into the microbial Tree of Life

Abstract

Ultrametric matrices appear in many domains of mathematics and science; nevertheless, they can be large and dense, making them difficult to store and manipulate, unlike large but sparse matrices. In this manuscript, we exploit that ultrametric matrices can be represented as binary trees to sparsify them via an orthonormal base change based on Haar-like wavelets. We show that, with overwhelmingly high probability, only an asymptotically negligible fraction of the off-diagonal entries in random but large ultrametric matrices remain non-zero after the base change; and develop an algorithm to sparsify such matrices directly from their tree representation. We also identify the subclass of matrices diagonalized by the Haar-like wavelets and supply a sufficient condition to approximate the spectrum of ultrametric matrices outside this subclass. Our methods give computational access to a covariance matrix model of the microbiologists’ Tree of Life, which was previously inaccessible due to its size, and motivate introducing a new wavelet-based (beta-diversity) metric to compare microbial environments. Unlike the established metrics, the new metric may be used to identify internal nodes (i.e. splits) in the Tree that link microbial composition and environmental factors in a statistically significant manner.

Interpretable metric learning in comparative metagenomics: The adaptive Haar-like distance

“…In this section, we outline our metric learning algorithm. First, we discuss the Haar-like wavelet [14] basis and its corresponding coordinate system, which gives rise to the Haar-like distance [15]. We then generalize this metric by introducing tunable weight parameters, leading to the adaptive Haar-like distance and corresponding kernel.…”

“…In particular, since the latter represent speciation events that group OTUs into clades, the Haar-like wavelets offer a basis for comparing clades of microorganisms as opposed to separate OTUs. So, assuming sample abundances correlate within the same clade across similar environments, projecting these onto the wavelets should elucidate relationships between microbial composition and environmental factors [15].…”

“…For any internal node v, i.e., v 2 I, denote by L(v) the set of leaves that descend from v. Further, v + and v − denote the left and right children descending from v, respectively. (In [15], these were denoted v0 and v1, respectively. )…”

“…Recent work showed that discrete Haar-like wavelets [14] could significantly pseudodiagonalize (i.e., sparsify) the phylogenetic covariance matrix of most large binary trees by changing basis to the wavelets. This motivated the introduction of a novel phylogenetic β-diversity metric known as the Haar-like distance [15]. This new metric may be regarded as a proxy for DPCoA; however, unlike DPCoA and UniFrac, it admits a simple decomposition in terms of the splits or bifurcations (i.e., internal nodes) of the reference phylogeny, which enables further interpretation and visualization of microbial sample distances in terms of differences between microbial clade abundances.…”

“…In this manuscript, we introduce a new phylogenetic β-diversity metric: the adaptive Haarlike distance, which is inspired by the recent Haar-like distance [15]. Taking a metric learning approach [27], our algorithm learns a data-dependent weighting of the most important phylogenetic relationships across a set of samples to discover robust representations of microbial abundance patterns.…”