Quantitative description of animal social behavior is informative for behavioral biologists and clinicians developing drugs to treat social disorders. Social interaction in a group of animals has been difficult to measure since behavior develops over long periods of time and requires tedious manual scoring which is subjective and often non-reproducible. Computer-vision systems that could measure complex social behavior automatically would have a transformative impact in biology. Here, we present a method for tracking group-housed mice individually as they freely interact over multiple days. Each mouse is bleach-marked with a unique fur pattern. The patterns are automatically learned by the tracking software and used to infer identities. Trajectories are analyzed to measure behavior, as it develops over days, beyond the range of acute experiments. We demonstrate how our system may be used to study the development of place preference, association and social relationships by tracking four mice continuously for five days. Our system enables accurate and reproducible characterization of wild-type mouse social behavior, and paves the way for high-throughput long-term observation of the effects of genetic, pharmacological and environmental manipulations.

In this review, we discuss the emerging field of computational behavioral analysis-the use of modern methods from computer science and engineering to quantitatively measure animal behavior. We discuss aspects of experiment design important to both obtaining biologically relevant behavioral data and enabling the use of machine vision and learning techniques for automation. These two goals are often in conflict. Restraining or restricting the environment of the animal can simplify automatic behavior quantification, but it can also degrade the quality or alter important aspects of behavior. To enable biologists to design experiments to obtain better behavioral measurements, and computer scientists to pinpoint fruitful directions for algorithm improvement, we review known effects of artificial manipulation of the animal on behavior. We also review machine vision and learning techniques for tracking, feature extraction, automated behavior classification, and automated behavior discovery, the assumptions they make, and the types of data they work best with.

No abstract

Recent developments in machine vision methods for automatic, quantitative analysis of social behavior have immensely improved both the scale and level of resolution with which we can dissect interactions between members of the same species. In this paper, we review these methods, with a particular focus on how biologists can apply them to their own work. We discuss several components of machine vision-based analyses: methods to record high-quality video for automated analyses, video-based tracking algorithms for estimating the positions of interacting animals, and machine learning methods for recognizing patterns of interactions. These methods are extremely general in their applicability, and we review a subset of successful applications of them to biological questions in several model systems with very different types of social behaviors.

Organisms that use vocal signals to communicate often modulate their vocalizations to avoid being masked by other sounds in the environment. Although some environmental noise is continuous, both biotic and abiotic noise can be intermittent, or even periodic. Interference from intermittent noise can be avoided if calls are timed to coincide with periods of silence, a capacity that is unambiguously present in insects, amphibians, birds, and humans. Surprisingly, we know virtually nothing about this fundamental capacity in nonhuman primates. Here we show that a New World monkey, the cotton-top tamarin (Saguinus oedipus), can restrict calls to periodic silent intervals in loud white noise. In addition, calls produced during these silent intervals were significantly louder than calls recorded in silent baseline sessions. Finally, average call duration dropped across sessions, indicating that experience with temporally patterned noise caused tamarins to compress their calls. Taken together, these results show that in the presence of a predictable, intermittent environmental noise, cotton-top tamarins are able to modify the duration, timing, and amplitude of their calls to avoid acoustic interference.

Mice (Mus musculus) form large and dynamic social groups and emit ultrasonic vocalizations in a variety of social contexts. Surprisingly, these vocalizations have been studied almost exclusively in the context of cues from only one social partner, despite the observation that in many social species the presence of additional listeners changes the structure of communication signals. Here, we show that male vocal behavior elicited by female odor is affected by the presence of a male audience -with changes in vocalization count, acoustic structure and syllable complexity. We further show that single sensory cues are not sufficient to elicit this audience effect, indicating that multiple cues may be necessary for an audience to be apparent. Together, these experiments reveal that some features of mouse vocal behavior are only expressed in more complex social situations, and introduce a powerful new assay for measuring detection of the presence of social partners in mice.

SUMMARY Auditory feedback is critical for the development and maintenance of speech in humans. In contrast, studies of nonhuman primate vocal production generally report that subjects show little reliance on auditory input. We examined the extent to which cotton-top tamarin (Saguinus oedipus) vocal production is sensitive to perturbation of auditory feedback by manipulating the predictability of presentation of a 1 s burst of white noise during the production of the species-specific contact call, the combination long call(CLC). We used three experimental conditions: the Begin condition, in which white noise was presented only during the first half of a recording session,the End condition, in which white noise was presented only in the last half,and the Random condition, in which each call had a 50% probability of receiving white noise playback throughout the recording session, making the auditory feedback unpredictable. In addition we recorded calls before and after the experimental series (Baseline condition) to determine whether any changes induced by modification of auditory feedback persisted. Results showed that playback of white noise during the production of the CLC produced changes in the temporal structure of the CLC: calls were shorter and had fewer pulses,indicating that modification of auditory feedback can interrupt vocal production. In addition, calls that received modified feedback were louder and had longer inter-pulse intervals than those that did not, consistent with an adaptive response to the masking effect of white noise playback. The magnitude of this compensatory effect and the interruption rate were both sensitive to whether the feedback modification occurred at the beginning or end of the experimental session: early feedback produced less interruption and more compensation. Finally, when auditory feedback modification was unpredictable,adaptive changes were observed in both calls that received modified feedback and those that received normal feedback, suggesting that tamarins can generate an expectation of noise playback and increase vocal amplitude in anticipation of masking.

The Lombard effect-an increase in vocalization amplitude in response to an increase in background noise-is observed in a wide variety of animals. We investigated this basic form of vocal control in the cotton-top tamarin (Saguinus oedipus) by measuring the amplitude of a contact call, the combination long call (CLC), while simultaneously varying the background noise level. All subjects showed a significant increase in call amplitude and syllable duration in response to an increase in background noise amplitude. Together with prior results, this study shows that tamarins have greater vocal control in the context of auditory feedback perturbation than previously suspected.