Discordance of Species Trees with Their Most Likely Gene Trees: A Unifying Principle

Abstract

A labeled gene tree topology that disagrees with a labeled species tree topology is said to be anomalous if it is more probable under a coalescent model for gene lineage evolution than the labeled gene tree topology that matches the species tree. It has previously been shown that as a consequence of short internal branches of the species tree, for every labeled species tree topology with five or more taxa, and for asymmetric four-taxon species tree topologies, an assignment of species tree branch lengths can be made which gives rise to anomalous gene trees (AGTs). Here, I offer an alternative characterization of this result—a labeled species tree topology produces AGTs if and only if it contains two consecutive internal branches in an ancestor–descendant relationship—and I provide a proof that follows from the change in perspective. The reformulation and alternative proof of the existence result for AGTs provide the insight that it is not merely short internal branches that generate AGTs, but instead, short internal branches that are arranged consecutively.

Statistical binning enables an accurate coalescent-based estimation of the avian tree

“…One advantage of such a phylogenomic approach is that it enables more data to be used in tree estimation (1). However, there is increasing evidence that loci can have conflicting evolutionary histories (so that their phylogenetic trees are topologically different) because of many biological causes, including incomplete lineage sorting (ILS) (2, 3), a process that is especially common in rapid radiations, characterized by a succession of short branches in the phylogenetic tree, such as is believed to have occurred in the avian and mammalian evolutionary lineages (4)(5)(6)(7)(8)(9). However, the standard phylogenetic estimation technique of concatenation, which concatenates alignments for individual loci into a combined data set called a supermatrix and then estimates the species tree from the supermatrix, can return incorrect species trees with high confidence in the presence of substantial ILS (10)(11)(12)(13).…”

Inferring rooted species trees from unrooted gene trees using approximate Bayesian computation

“…Simmons and Gatesy (2015) argue that the anomaly zone, in which the most probable gene tree has a different topology from that of the species tree, does not apply to the rooting of the angiosperms because there is only one short branch leading to the Amborella-Nuphar clade, whereas the anomaly zone typically occurs when there are two consecutive branches on a path from the root to the tips (Degnan, 2013; Rosenberg, 2013). Although this is typical of species trees in the anomaly zone, a species tree with only one short branch can still be in the anomaly zone if the less basal branch can be indefinitely long if the more basal branch is sufficiently short (Degnan and Rosenberg, 2006).…”

Consistency and inconsistency of consensus methods for inferring species trees from gene trees in the presence of ancestral population structure

“…In addition to methods that are consistent under the standard multispecies coalescent, we considered two methods, Democratic Vote and MDC, that are inconsistent in this basic setting [Degnan and Rosenberg, 2006, Than and Rosenberg, 2011, Rosenberg, 2013]. Adding ancestral population structure does not eliminate the inconsistency, and both methods are misleading in the model with ancestral structure.…”

“…Democratic Vote estimates a species tree topology using the most frequently occurring gene tree topology, or uniquely favored topology, in a sample of gene trees [Degnan and Rosenberg, 2009]. Discordant gene tree topologies that are more probable than the matching topology have been termed “anomalous gene trees” (AGTs), and the space of branch lengths in which AGTs arise has been termed the “anomaly zone” [Degnan and Rosenberg, 2006, Rosenberg, 2013]. Owing to AGTs, and because of gene tree discordance more generally, it is difficult for consensus methods to achieve statistical consistency [Degnan et al, 2009].…”

“…We introduce a specific scenario and use it to prove that in part of the parameter space of our model, Democratic Vote [Degnan and Rosenberg, 2006, 2009, Rosenberg, 2013], STAR [Liu et al, 2009], STEAC [Liu et al, 2009], R * Consensus [Bryant, 2003, Degnan et al, 2009], Rooted Triple Consensus [Ewing et al, 2008], Minimize Deep Coalescences [MDC; Maddison, 1997, Maddison and Knowles, 2006, Than and Nakhleh, 2009, Nakhleh, 2013], and Majority-Rule Consensus [Degnan et al, 2009] are misleading in that the probability that a consensus tree contains a clade not on the species tree approaches 1 as the number of loci goes to ∞. Note that we only explore a subset of the available methods for inferring species trees from gene trees, and that the asymptotic properties of other methods could be investigated using similar approaches.…”