Automated annotation of birdsong with a neural network that segments spectrograms

Abstract

Songbirds provide a powerful model system for studying sensory-motor learning. However, many analyses of birdsong require time-consuming, manual annotation of its elements, called syllables. Automated methods for annotation have been proposed, but these methods assume that audio can be cleanly segmented into syllables, or they require carefully tuning multiple statistical models. Here we present TweetyNet: a single neural network model that learns how to segment spectrograms of birdsong into annotated syllables. We show that TweetyNet mitigates limitations of methods that rely on segmented audio. We also show that TweetyNet performs well across multiple individuals from two species of songbirds, Bengalese finches and canaries. Lastly, we demonstrate that using TweetyNet we can accurately annotate very large datasets containing multiple days of song, and that these predicted annotations replicate key findings from behavioral studies. In addition, we provide open-source software to assist other researchers, and a large dataset of annotated canary song that can serve as a benchmark. We conclude that TweetyNet makes it possible to address a wide range of new questions about birdsong.

Shared mechanisms of auditory and non-auditory vocal learning in the songbird brain

“…We did so to account for potential circadian effects on song production. We also reassessed our results (shown in Figure 2 ) by analyzing only syllable renditions produced between 6 PM and 8 PM using new methods for automated labeling of song syllables ( Cohen et al, 2022 ). We found no statistically significant difference in learning magnitudes between the two forms of analysis ( Figure 2—figure supplement 7a , 0.167 < P boot < 0.951 on all days of training).…”

The impacts of fine-tuning, phylogenetic distance, and sample size on big-data bioacoustics

“…We arbitrarily chose a subset of recordings to annotate from each species, starting with Zonotrichia leucophrys which had the largest dataset, and moving down to Calypte anna which had the smallest dataset. As few as three minutes of recording is sufficient for the TweetyNet algorithm to perform accurately [50], so we ensured that all species had at least 180 seconds worth of annotated syllables. We then added more recordings for species with more data to investigate the influence of sample size.…”

“…With respect to bioacoustics, an extractive algorithm could, for example, segment out syllables within vocalizations. A recently developed application, TweetyNet, was released to perform just this task [50,85,86] using deep learning via ANNs. Specifically, TweetyNet uses convolutional and recurrent ANNs.…”

“…Though the sample size does not appear to impact accuracy, it is an important consideration when trying to decide how to best extract data from species. With sufficiently large sample sizes (TweetyNet suggests 3 minutes of annotation, [50]), techniques like convolutional neural networks will work effectively to learn and parse the diversity of syllables and sounds present in these data. However, for some taxa the available data are not sufficient for these types of learning.…”

“…Although challenges remain with respect to scalability, computational efficiency, and how to handle depauperate data [42], deep learning is one of the most powerful analytical tools in the modern researcher’s toolbox, particularly when human knowledge is lacking, or datasets are too large to be workable by traditional means. In the context of ecology and evolutionary biology, there have been many recent applications of both shallow and deep machine learning, including population genetics and phylogeography [e.g., 46, 47], bioacoustics [e.g., 48, 49, 50], species classification [e.g., 51], phylogenetics [e.g., 52, 53], sequencing and genomics [e.g., 54, 55], and phenotypic analyses and morphometrics [e.g., 56, 57]. Neural networks and support vector machines tend to be the most applied algorithms towards these analyses.…”