Consistency and inconsistency of consensus methods for inferring species trees from gene trees in the presence of ancestral population structure

Abstract

In the last few years, several statistically consistent consensus methods for species tree inference have been devised that are robust to the gene tree discordance caused by incomplete lineage sorting in unstructured ancestral populations. One source of gene tree discordance that has only recently been identified as a potential obstacle for phylogenetic inference is ancestral population structure. In this article, we describe a general model of ancestral population structure, and by relying on a single carefully constructed example scenario, we show that the consensus methods Democratic Vote, STEAC, STAR, R* Consensus, Rooted Triple Consensus, Minimize Deep Coalescences, and Majority-Rule Consensus are statistically inconsistent under the model. We find that among the consensus methods evaluated, the only method that is statistically consistent in the presence of ancestral population structure is GLASS/Maximum Tree. We use simulations to evaluate the behavior of the various consensus methods in a model with ancestral population structure, showing that as the number of gene trees increases, estimates on the basis of GLASS/Maximum Tree approach the true species tree topology irrespective of the level of population structure, whereas estimates based on the remaining methods only approach the true species tree topology if the level of structure is low. However, through simulations using species trees both with and without ancestral population structure, we show that GLASS/Maximum Tree performs unusually poorly on gene trees inferred from alignments with little information. This practical limitation of GLASS/Maximum Tree together with the inconsistency of other methods prompts the need for both further testing of additional existing methods and development of novel methods under conditions that incorporate ancestral population structure.

Statistical inconsistency of the unrooted minimize deep coalescence criterion

“…This phenomenon, also called incomplete lineage sorting, is modeled by the multispecies coalescent, which makes probabilistic predictions for the probabilities of different gene tree topologies to be observed in a sample of gene trees [3][4][5]. Although other sources of gene tree heterogeneity are possible, such as gene duplication and loss, hybridization, recombination within genes, and ancient population structure [1,2,6,7] deep coalescence is thought to be quite common for species that underwent rapid radiations [8,9] and is often used to infer species relationships from gene trees [2,10,11].…”

Modeling Hybridization Under the Network Multispecies Coalescent

“…In particular, Slatkin and Pollack (2008) demonstrate that the most probable topology does not necessarily match the species tree topology, even with three taxa. Consequently, many popular two-stage methods of inferring species trees can be inconsistent when there is ancestral population structure, meaning that they can fail to recover the species tree even with arbitrarily large numbers of loci (DeGiorgio and Rosenberg 2016 5 with X = e −t 2 , Y = e −t 1 , and where = 1/3 is the probability that lineages go to the left Network/tree from Fig. 5 Gene tree (a give rise to similar patterns is that population structure might give the appearance of distinct species when the MSC is used for species delimitation without accounting for population structure (Sukumaran and Knowles 2017).…”

“…If the proportions of these two discordant trees differ significantly from each other, then this is evidence against the null hypothesis of the MSC, and this is sometimes used as evidence against the MSC as adequately describing the data (Degnan and Rosenberg 2009;Ané 2010;Chung and Ané 2011;Song et al 2012). However, the MSC + population structure and MSC + hybridization models can both predict asymmetries in the two discordant topologies (Slatkin and Pollack 2008;Meng and Kubatko 2009;DeGiorgio and Rosenberg 2016). Consequently, distinguishing these two models can be quite difficult.…”

Maximum likelihood estimation of species trees from gene trees in the presence of ancestral population structure

“…There are several hypothesized examples of structured ancestral species (e.g., Garrigan et al 2005;Thalmann et al 2007;White et al 2009), and genomic signatures of ancestral structure have been uncovered in a number of diverse lineages, including mouse (White et al 2009) and yeast (Yu et al 2012). Still, many methods for inferring species trees from multilocus data are not robust to ancestral structure, and can be proven to be positively misleading (DeGiorgio and Rosenberg 2016). DeGiorgio and Rosenberg (2016) demonstrate that exceptions to this rule are the maximum likelihood estimators of the species tree implemented in GLASS (Mossel and Roch 2010), STEM (Kubatko et al 2009), andMaximum Tree (Liu et al 2010c); however, these algorithms underperform on empirical data when gene trees are inferred rather than known.…”

“…Still, many methods for inferring species trees from multilocus data are not robust to ancestral structure, and can be proven to be positively misleading (DeGiorgio and Rosenberg 2016). DeGiorgio and Rosenberg (2016) demonstrate that exceptions to this rule are the maximum likelihood estimators of the species tree implemented in GLASS (Mossel and Roch 2010), STEM (Kubatko et al 2009), andMaximum Tree (Liu et al 2010c); however, these algorithms underperform on empirical data when gene trees are inferred rather than known.…”