Testing peatland testate amoeba transfer functions: Appropriate methods for clustered training-sets

Abstract

45 46Transfer functions are widely used in palaeoecology to infer past environmental 47conditions from fossil remains of many groups of organisms. In contrast to 48 traditional training-set design with one observation per site, some training sets, 49 including those for peatland testate amoeba-hydrology transfer functions, have a 50 clustered structure with many observations from each site. Here we show that this 51 clustered design causes standard performance statistics to be overly optimistic. 52Model performance when applied to independent data sets is considerably weaker 53 than suggested by statistical cross-validation. We discuss the reasons for these 54 problems and describe leave-one-site-out cross-validation and the cluster bootstrap 55 as appropriate methods for clustered training sets. Using these methods we show 56 that the performance of most testate amoeba-hydrology transfer functions is worse 57 than previously assumed and reconstructions are more uncertain. One motivation for developing clustered training-sets is the presence within 74 each site of substantial environmental gradients, which may be large relative to the 75 differences between sites. This contrasts with the traditional one observation per 76 site training-set where typically the environmental variable (e.g. lake-pH) is assumed 77 to be spatially homogeneous at each site. Standard methods for assessing the 78 performance of transfer functions assume that the observations are independent 79 and are thus inappropriate for clustered data. Lack of independence between 80 observations, either because of spatial autocorrelation or a clustered design, will 81 cause performance statistics to be over-optimistic (Telford and Birks, 2005 (Fig. 1). 97To provide an independent estimate of transfer function performance, we 98 apply five transfer functions to all comparable independent datasets with 99 appropriate corrections for taxonomic and methodological differences (Appendix I). 100 Table 1 shows that most transfer functions perform worse than suggested by leave-101 one-out (LOO) cross-validation when applied to independent data. Methodological 102 explanations for the poor model performance can largely be excluded. Differences in 103 time-discrete water-table measurements cannot explain the differences in rank-104 order shown by Spearman's ρ. Any differences in sample preparation and analysis, or 105 residual taxonomic biases cannot explain poor performance where these are closely 106harmonised (e.g. Polish data). Performance is particularly poor for two datasets from 107 Typically, transfer function model performance is assessed by either leave-117 one-out (LOO) or bootstrap cross-validation. In LOO, one observation at a time is 118 omitted from the training-set of size n and the environmental value predicted using 119 the remaining n-1 observations. For clustered data, this can be extended to leave-120 one-site-out cross-validation (LOSO), where data from one site is omitted from the 121 training set, and data from the remaining m...

Comparing regional and supra-regional transfer functions for palaeohydrological reconstruction from Holocene peatlands

“…The validity of 'one-shot' WTD measurements has been debated (Bobrov et al, 1999;Booth, 2008), however provided extreme pre-sampling weather conditions are avoided (prolonged rain, drought) this has been regarded as adequate for transfer function development (Booth, 2008;Charman et al, 2007;Payne et al, 2011;). Samples were retained in sealed 'Ziploc' type bags, returned to the laboratory and stored at 4°C prior to analysis.…”

“…Although the majority of palaeohydrological studies have been carried out on ombrotrophic raised bogs, capturing taxa associated with other bog types may be critical to understanding bog development trajectories. We included a variety of bog types in site selection to 1) ensure the inclusion of as many modern analogues as possible; and 2) sample full environmental gradients as recommended by Payne et al (2011). However, we acknowledge that this sampling strategy cannot account for the true complexity of the autogenic and allogenic processes operating within peatlands (cf.…”

“…Such taxa may be important for interpreting bog development trajectories including pathways to ombrotrophy. Payne et al (2011) offer a number of recommendations to improve clustered training sets that we have followed here; 1) a large number of sites should ideally be sampled to reduce autocorrelation problems; 2) sites should be similar in terms of climate and vegetation; 3) there should be sufficient diversity in the sample sites to ensure there are good analogues for all fossil samples in the training set; 4) the full environmental gradient is sampled at each site. However, the encompassment of the full hydrological gradient at each site is somewhat weakened by the need to exclude outlier samples to achieve greater predictive power.…”

Exploring the diversity and ecology of testate amoebae in West Siberian peatlands

“…The links between TA communities and 'wetness' are well-established in the literature (Smith et al 2008). Numerous previous correlative studies have found strong links beTestate Amoebae in West Siberian Peatlands 67 tween TA communities and soil wetness, often explaining more species variance than other variables (Payne et al 2012a). A substantial majority of these studies have considered water table depth (WTD) as the hydrological metric.…”

“…This trend has been particularly driven by increasing interest from palaeoecologists in using the preserved shells of TA as palaeoenvironmental proxies for the reconstruction of long-term change in peatland water table (Charman 1999). To calibrate these records more than thirty studies of modern communities have been undertaken with the aim to produce species-environment models ('transfer functions') to allow palaeoecological data to be interpreted in quantitative terms (Payne et al 2012aTsyganov et al 2016). There is now increasing interest in combining these individual studies into larger datasets which can be used to draw broader conclusions (Amesbury et al 2016, Booth 2008.…”