Distribution of gene tree histories under the coalescent model with gene flow

Abstract

We propose a coalescent model for three species that allows gene flow be-23 tween both pairs of sister populations. The model is designed to analyze multilocus genomic 24 sequence alignments, with one sequence sampled from each of the three species. The model 25 is formulated using a Markov chain representation, which allows use of matrix exponentia-26 tion to compute analytical expressions for the probability density of gene tree genealogies. 27 The gene tree history distribution as well as the gene tree topology distribution under this 28 coalescent model with gene flow are then calculated via numerical integration. We analyze 29 the model to compare the distributions of gene tree topologies and gene tree histories for 30 species trees with differing effective population sizes and gene flow rates. Our results suggest 31 conditions under which the species tree and associated parameters are not identifiable from 32 the gene tree topology distribution when gene flow is present, but indicate that the gene 33 tree history distribution may identify the species tree and associated parameters. Thus, the 34 gene tree history distribution can be used to infer parameters such as the ancestral effective 35 population sizes and the rates of gene flow in a maximum likelihood (ML) framework. We 36 conduct computer simulations to evaluate the performance of our method in estimating these 37 parameters, and we apply our method to an Afrotropical mosquito data set (Fontaine et al., 38 2015) to demonstrate the usefulness of our method for the analysis of empirical data. 39 40 maximum likelihood, speciation. 41 42 gruence between a gene tree and the species tree for the same set of species (Maddison, 43 1997). Incomplete lineage sorting (also called deep coalescence) has long been recognized to 44 be one of the major causes of variation in gene trees across a genome (Pamilo and Nei, 1988; 45 Takahata, 1989). Another important factor leading to discord between gene trees and the 46 species tree is gene flow between populations following speciation (Maddison, 1997; Leaché 47 et al., 2013; Degnan et al., 2012). With some exceptions (noted below), these two pro-48 cesses have been studied in isolation. When carrying out phylogenetic analyses for species 49 that are substantially divergent, ignoring gene flow following speciation may not bias the 50 resulting estimates. However, with the advent of large-scale genomic data sets that allow 51 study of evolutionary relationships among closely related populations or species, the neces-52 sity of simultaneously examining these factors is becoming increasingly apparent (Eckert 53 and Carstens, 2008; Leaché et al., 2013; Huang et al., 2014). In particular, since gene flow 54 may easily occur between sister taxa following speciation, even in the presence of incomplete 55 lineage sorting (Yu et al., 2011), it is necessary to incorporate these processes simultaneously 56 into models used to analyze data for closely related species or populations. 57Degnan and Sal...

The Effect of Gene Flow on Coalescent-based Species-Tree Inference

“…While this scenario gives insight into what conditions might produce anomalous behavior, it is not actually a requirement that the rate of gene flow from D to C be greater than the rate from C to D (i.e., that m 1 > m 2 ). In [24], the authors analyzed a model with equal rates of gene flow, (i.e., where m 1 = m 2 ) and showed the concordant triple frequency could be made very close to 1/3. In fact, the following proposition shows that even with this restriction, the concordant triple frequency can be made to approach 1/9.…”

“…Isolation-with-migration (IM) models have a long history in the population genetics literature [9,8,25], and have more recently been considered in the context of species tree estimation [26,24]. Here we consider a more general version of the IM model than that presented in [24]. Figure 1a shows a species tree with three species, labeled A, B, and C, with topology ((A, B), C).…”

“…While several studies have examined the impact of gene flow on the performance of specieslevel phylogenetic inference (e.g., [6,4,16]), analytic analyses of the potential impact of gene flow are limited, despite the development of mathematical tools that could facilitate such study (e.g., [10,1,24]). Recently, Tian and Kubatko (2016) considered a three-species IM (isolationwith-migration) model [9,8,25] that allowed gene flow between sister species, and derived the probability distribution of gene tree topologies and gene tree histories under this model.…”

“…Their work generalized the work of Zhu and Yang (2012), who considered a similar model but with gene flow allowed only between the terminal sister taxa. Both Zhu and Yang (2012) and Tian and Kubatko (2016) used their results to estimate model parameters via maximum likelihood. Zhu and Yang (2012) considered sequence data arising from the model, while Tian and Kubatko (2016) considered the observed distribution of gene tree histories.…”

“…Here, we consider the distribution of gene tree topologies for a three-species IM model that generalizes that considered by Tian and Kubatko (2016) to include time periods with and without gene flow between both pairs of sister species. We further consider the effect of variation in population sizes and migration rates on this distribution.…”