Analysis of Ultrasonic Vocalizations from Mice Using Computer Vision and Machine Learning

Abstract

Mice emit ultrasonic vocalizations (USV) to transmit socially-relevant information. To detect and classify these USVs, here we describe the development of VocalMat. VocalMat is a software that uses image-processing and differential geometry approaches to detect USVs in audio files, eliminating the need for user-defined parameter tuning. VocalMat also uses computational vision and machine learning methods to classify USVs into distinct categories. In a dataset of >4,000 USVs emitted by mice, VocalMat detected more than >98% of the USVs and accurately classified ≈86% of USVs when considering the most likely label out of 11 different USV types. We then used Diffusion Maps and Manifold Alignment to analyze the probability distribution of USV classification among different experimental groups, providing a robust method to quantify and qualify the vocal repertoire of mice. Thus, VocalMat allows accurate and highly quantitative analysis of USVs, opening the opportunity for detailed and high-throughput analysis of this behavior. * Present address: Interdepartmental Neuroscience Program, Yale School of Medicine, New Haven -CT, United States of America 2 Results Detection of mouse USVs using imaging processingVocalMat uses multiple steps to analyze USVs from vocalizing mice in audio files (see Figure 1A for the general workflow). Initially, the audio recordings are converted into high-resolution spectrograms through a short-time Fourier transformation (see Methods and Materials). The resulting spectrogram consists of a matrix, wherein each element corresponds to an intensity value (power spectrum represented in decibels) for each time-frequency component. The spectrogram is then analyzed in terms of its time-frequency plane, where high-intensity values are represented by brighter pixels in a gray-scale image ( Figure 1B). The gray-scale image undergoes contrast enhancement and adaptive thresholding for binarization (see Methods and materials). The segmented objects are further refined via morphological operations ( Figure 1C and Figure S1), thus resulting in a list of segmented blobs (hereafter referred to as USV candidates) with their corresponding spectral features ( Figure 1D).This list of USV candidates may contain noise (i.e., detected particles that are not part of any USV) and multiple candidates that correspond to the same USV. To address this, a minimum of 10 ms interval between two successive and distinct syllables is assumed based on experimental observations [9]. To reduce the amount of data stored for each USV, the features extracted from detected candidates are represented by a mean frequency and intensity every 0.5 ms. The means are calculated for all the individual candidates, including the ones overlapping in time, hence preserving relevant features such as duration, frequency, intensity, and harmonic components ( Figure 1D).Harmonic components are also referred to as nonlinear components or composite [31,30]. Here, we did not consider harmonic components as a different syllable, but rather as an extra fea...

Deficiency of the paternally inherited geneMagel2alters the development of separation-induced vocalization and maternal behavior in mice

“…Mouse pups emit USVs of distinct classes-i.e., syllable types. Thus, the emission of different syllable types could explain the discrete changes in the spectro-temporal features of USVs in Magel2 m+/p− (Figure 3A) 33,37 . We used machine learning to automatically categorize each USV into one of eleven syllable types based on the morphology of the main component of the vocalization.…”

“…In addition to the rate of separation-induced vocalizations, the spectral features of the USVs also correlate with altered maternal care 36 . To test the extent to which Magel2 deficiency affects the spectral features of USVs across ages, we used two-way ANOVA to analyze the frequency characteristics (pitch) and duration (Figure 2A) of USVs 33 . We found significant effects of genotype and age for maximal frequency (genotype: F1, 182 = 20.38, P < 10 -4 ; age: F3, 182 = 2.79, P = 0.04; genotype x age: F3, 182 = 6.45, P = 0.0004; Figure 2B) and bandwidth (genotype: F1, 182 = 9.79 P = 0.002; age: F3, 182 = 6.94, P = 0.0002; genotype x age: F3, 182 = 5.88, P = 0.0007; Figure 2E).…”

“…As stated above, the vocal analysis pipeline outputs the probability for each USV to be classified as each of the eleven syllable types (P1, P2, P3, … P11; Figure 3A-B). This distribution of probabilities allows the qualitative and quantitative comparison of the vocal classification among groups 33 . By considering the distribution of probabilities to classify each USV, it is possible to estimate how similar the vocal repertoire of one group of mice is to another group.…”

“…By considering the distribution of probabilities to classify each USV, it is possible to estimate how similar the vocal repertoire of one group of mice is to another group. To compare the vocal repertoire of mice across all ages studied, we used diffusion maps-a dimensionality reduction technique that decreases the number of dimensions of the probability distribution from eleven classes to three dimensions in a Euclidean space (Figure 4A) 33 . Using pairwise comparisons (Figure 4B), we estimated the similarity between the vocal repertoire of mice of was not certified by peer review) is the author/funder.…”

“…We performed these studies at postnatal days 6, 8, 10, and 12, since it is during this phase of mouse development that the peak expression of separation-induced vocalizations typically occurs 15,31,32 . We then used VocalMat 33 , a software developed by our group, to perform quantitative analysis of mouse vocal behavior. Our analysis shows that the deficiency of Magel2 in mice impairs the expression of separation-induced vocalizations, a behavior that supports pups' growth and development.…”