A deep neural network for high‐throughput measurement of functional traits on museum skeletal specimens

Abstract

Increasingly, natural history museum collections are being used to generate large‐scale morphological datasets to address a range of macroecological and macroevolutionary questions. One challenge to this approach is that large numbers of individuals either from a single species or from taxonomically broad sets of species may be necessary to characterize morphology at the relevant spatial, phylogenetic or temporal scales. We present ‘Skelevision’, a method for rapidly handling, photographing and measuring skeletal specimens with a computer vision approach that uses a deep neural network to segment the photographs of specimens into individual bones, and identify and measure functional aspects of those bones. We demonstrate the scale of what is feasible with Skelevision by estimating 11 functional traits from 11 different bones for 12,450 bird skeletal specimens spanning 1,882 species of passerines (~32% of all passerine diversity). We quantify the accuracy of Skelevision estimates by comparing them to handmade measurements for 174 specimens from 115 species across 79 genera that span 59 families. Skelevision is precise, with a mean standard deviation of 0.86 mm for repeated independent measurements of individual bones, and is extremely accurate, with a mean RMSE of 0.89 mm across all traits when compared to handmade measurements. There is minimal phylogenetic signal in the measurement error (mean Pagel's λ across traits = 0.13), and Skelevision estimates are robust to variation in the degree to which specimens remain articulated. This approach has several important advantages over traditional methods for building large‐scale morphological datasets (e.g. measurements from long‐term field‐based operations or handmade measurements of museum specimens). First, measuring new specimens only requires the collection of photographs, which can then be measured automatically, and effectively instantaneously, with the neural network. This is a significant departure from the time and skill required to measure skeletal specimens by hand. Second, the measurements are repeatable. Third, even as the dataset of photographed specimens expands, the amount of annotation data needed to measure new traits on all of the photographed specimens using the neural network will remain fixed and can be done without re‐capturing images.

Fossil image identification using deep learning ensembles of data augmented multiviews

“…Removing colour may result in greater differences (good or bad) for other potential fossil groups and thus requires caution. Conversion to skeletonized images is also useful as it helps represent the morphological structure of the fossils and thus serves as a feature extraction (Saha et al, 2016; Weeks et al, 2023). For shell‐forming organisms like fusulinids, the topology of their shell, such as the number and size of chambers and the manner of spinning and coiling, is sufficient to provide a great deal of information for their identification and classification (Ross & Ross, 1991; Sheng et al, 1988; Vachard et al, 2010).…”

“…the fossils and thus serves as a feature extraction (Saha et al, 2016;Weeks et al, 2023). For shell-forming organisms like fusulinids, the topology of their shell, such as the number and size of chambers and the manner of spinning and coiling, is sufficient to provide a great deal of information for their identification and classification (Ross & Ross, 1991;Sheng et al, 1988;Vachard et al, 2010).…”

Leveraging natural history collections to understand the impacts of global change

“…Using Mothra, they found that, for most of the species they examined, butterfly size is increasing with temperature. Weeks et al (2023) introduce Skelevision, a method for rapidly handling, photographing, and measuring skeletal specimens. They use Skelevision to estimate 11 traits from 11 different bones from 12,450 specimens of 1882 passerine birds, with a handling time of ~1 min per specimen.…”

Skeletal morphology of bird wings is determined by thermoregulatory demand for heat dissipation in warmer climates

“…Please contact BCW for access to the data. All code for generating bone measurement data is publicly available ( 18 )…”