Imputation of incomplete large‐scale monitoring count data via penalized estimation

Abstract

1. In biodiversity monitoring, large datasets are becoming more and more widely available and are increasingly used globally to estimate species trends and conservation status. These large-scale datasets challenge existing statistical analysis methods, many of which are not adapted to their size, incompleteness and heterogeneity. The development of scalable methods to impute missing data in incomplete large-scale monitoring datasets is crucial to balance sampling in time or space and thus better inform conservation policies. 2. We developed a new method based on penalized Poisson models to impute and analyse incomplete monitoring data in a large-scale framework. The method allows parameterization of (a) space and time factors, (b) the main effects of predictor covariates, as well as (c) space-time interactions. It also benefits from robust statistical and computational capability in large-scale settings.3. The method was tested extensively on both simulated and real-life waterbird data, with the findings revealing that it outperforms six existing methods in terms of missing data imputation errors. Applying the method to 16 waterbird species, we estimated their long-term trends for the first time at the entire North African scale, a region where monitoring data suffer from many gaps in space and time series.4. This new approach opens promising perspectives to increase the accuracy of species-abundance trend estimations. We made it freely available in the r package

Treating gaps and biases in biodiversity data as a missing data problem

“…Early approaches used chain indices or route regression (Ter Braak et al, 1992) or the Underhill index, using an expectation-maximisation algorithm (Underhill & Prysjones, 1994) designed for waterbirds (Rehfisch et al, 2003). A range of further model-based approaches have been developed that fill data gaps using mean effects of site and year, e.g., to fill annual gaps using TRIM/birdSTATs, commonly used for bird indices (Lehikoinen et al, 2016); or using splines e.g., to fill seasonal gaps in butterfly data (Schmucki et al, 2016;Dennis et al, 2016) or using ecological covariates (Dakki et al, 2021). A Bayesian framework is especially useful for dealing with missing values in the response since they are naturally imputed with a full probability distribution during model fitting.…”

Technical report: Impact of evaluation metrics and sampling on the comparison of machine learning methods for biodiversity indicators prediction

“…In previous work aiming to evaluate prediction methods for biodiversity data [2], authors of this report have faced the challenge of designing fair evaluation procedures, in order not to bias the conclusions, and thus provide sound and reliable recommendations for practitioners. Indeed, designing robust cross-validation (CV) evaluation metrics proved difficult, with different approaches yielding different results.…”

“…LORI The function lori of the package lori [2] uses an imputation method whose procedure resembles that of glmnet. The main difference is that, in this model, there are interaction parameters.…”

Analysing bird population trends from monitoring data with highly structured sampling designs

“…Yet, designed for monitoring schemes with almost-yearly surveys per site by the same observer, TRIM reaches its limits with substantial turnover in survey sites between years leading to a fraction of missing values exceeding approx. 60 % (van Strien et al 2001, Bogaart et al 2020, Dakki et al 2021). Moreover, TRIM is restricted to categorical covariates, requiring climate or landscape composition covariates to be transformed into categories (Bogaart et al 2020).…”