Reply to Mafessoni and Prüfer: Inferences with and without singleton site patterns

Abstract

Mafessoni and Prüfer (1) (MP) make three points: (i) Low variation among Neanderthal genomes implies a small population, (ii) Neanderthal-Denisovan divergence is small relative to archaic-modern, and (iii) an analysis including singleton site patterns (in which the derived allele appears only once) supports a small Neanderthal population and a more recent NeanderthalDenisovan separation.Point i assumes that sequenced Neanderthals are representative of all Neanderthals, yet samples come primarily from the north, where DNA preserves well. The global population may have been larger if some unsampled populations were distantly related to those sampled. Point ii is subsidiary to point iii, because divergences can be calculated from site pattern frequencies. We therefore focus on point iii.Our previous estimates (2) were biased, because our modern human data excluded invariant nucleotide sites (3). Mutations in an archaic lineage appear only if that lineage introgressed into moderns, leading to an undercount of pattern nd.We now analyze the four genomes studied by MP. Because these have high coverage and are not restricted to variable sites they do not introduce bias. We fit two models, one excluding singleton site patterns [as in our previous publication (2)] and one including them (as MP advocate).The left side of Fig. 1 presents results for the model without singletons. Our estimate of T ND , the Neanderthal-Denisovan separation time, is only slightly smaller than T XYND , the archaic-modern separation. This supports an early separation of the two archaics. Parameter 2N ND measures the haploid size of the population ancestral to Neanderthals and Denisovans. This estimate is small, implying an early bottleneck among archaics. The estimate of 2N N , the Neanderthal population size, is smaller than in our previous publication (2) but still larger than current estimates. These results are broadly consistent with those of our previous publication (2).In their with-singleton analysis MP use genetic data to estimate the ages of the two fossils. We do the same in our own with-singleton analysis, by defining these ages as free parameters. The results (Fig. 1, Right) are more similar to those of MP. Compared with the without-singleton analysis, the Neanderthal population is smaller, the ancestral archaic bottleneck is less severe, and the two archaics separate more recently. Nonetheless, our separation time (632 kya) is still much older than that of MP (459 kya).We are skeptical of the with-singleton analysis, because it implies that the Denisovan fossil lived only 4 kya. This is probably because site pattern d (derived allele only in Denisovan) is 10% more common than pattern n (only in Neanderthal). This may reflect diagenic changes in the Denisovan genome or hyperarchaic admixture into Denisova (4). Neither our models nor that of MP allows for this, so all may introduce bias. Residuals (Fig. 2) reinforce the view that something is missing from all three models.We favor the without-singleton analysis, which implies a fairl...

Neanderthal-Denisovan ancestors interbred with a distantly-related hominin

“…Mafessoni and Prüfer [15] pointed out that introducing singletons led to different results. In response Rogers et al [17] agreed, but also observed that the with-singleton analysis implied that the Denisovan fossil was only 4000 years old-a result that is plainly wrong. Furthermore, a residual analysis showed that neither of the models under discussion in 2017 fit the data very well [17].…”

“…In response Rogers et al [17] agreed, but also observed that the with-singleton analysis implied that the Denisovan fossil was only 4000 years old-a result that is plainly wrong. Furthermore, a residual analysis showed that neither of the models under discussion in 2017 fit the data very well [17]. Something was apparently missing from both models-but what?…”

“…3 shows the residuals from this model, using the new data. Several are far from zero, suggesting that something is missing from the model [17].…”

Neanderthal-Denisovan ancestors interbred with a distantly related hominin

“…Mafessoni and Prüfer (13) pointed out that introducing singletons led to different results. In response, Rogers et al (16) agreed, but also observed that the with-singleton analysis implied that the Denisovan fossil was only 4000 years old-a result that is plainly wrong. Furthermore, a residual analysis showed that neither of the models under discussion in 2017 fit the data very well (16) Our results shed light on the early portion of the middle Pleistocene, about 600 ka ago, when large-brained hominins appear in the fossil record of Europe along with Acheulean stone tools.…”

“…3 shows the residuals from this model, using the new data. Several are far from zero, suggesting that something is missing from the model (16).…”