JAABA: interactive machine learning for automatic annotation of animal behavior

Kabra, Mayank; Robie, Alice A.; Rivera-Alba, Marta; Branson, Steven; Branson, Kristin

doi:10.1038/nmeth.2281

Cited by 528 publications

(607 citation statements)

References 13 publications

Supporting

Mentioning

581

Contrasting

Unclassified

Order By: Relevance

“…Alternatively, this invariance can be encoded into the features. Following prior art [19,29,20] we have implemented a tool that tracks individual flies and segments them into body, wing, and leg pixels, which are parameterized further by fitting an oriented ellipse to the body component and line segments to the wing components. From the tracking output we derive a set of features that are designed to be invariant of the absolute position and orientation of a fly, and relate its pose to that of the other fly.…”

Section: Feature Representationmentioning

confidence: 99%

“…We implemented three variants of the above approaches, specifically comparing a sliding window SVM detector against two structured output SVM detectors, expecting the latter to improve frame-wise consistency and better capture structured actions. For reference, we compare our results with the methods described in [20] and [1] and with the performance of trained novice annotators.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Detecting Social Actions of Fruit Flies

Eyjolfsdottir

Branson

Burgos-Artizzu

et al. 2014

Lecture Notes in Computer Science

105

156

View full text Add to dashboard Cite

Abstract. We describe a system that tracks pairs of fruit flies and automatically detects and classifies their actions. We compare experimentally the value of a frame-level feature representation with the more elaborate notion of 'bout features' that capture the structure within actions. Similarly, we compare a simple sliding window classifier architecture with a more sophisticated structured output architecture, and find that window based detectors outperform the much slower structured counterparts, and approach human performance. In addition we test our top performing detector on the CRIM13 mouse dataset, finding that it matches the performance of the best published method. Our Fly-vs-Fly dataset contains 22 hours of video showing pairs of fruit flies engaging in 10 social interactions in three different contexts; it is fully annotated by experts, and published with articulated pose trajectory features.

show abstract

Section: Feature Representationmentioning

confidence: 99%

mentioning

confidence: 99%

Detecting Social Actions of Fruit Flies

Eyjolfsdottir

Branson

Burgos-Artizzu

et al. 2014

Lecture Notes in Computer Science

105

156

View full text Add to dashboard Cite

show abstract

“…Recently several techniques have been developed to track social behaviors in animals with rigid exoskeletons, such as the fruit fly Drosophila, which have relatively few degrees of freedom in their movements (13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23). These techniques have had a transformative impact on the study of social behaviors in that species (2).…”

mentioning

confidence: 99%

Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning

Hong

Kennedy

Burgos-Artizzu

et al. 2015

Proc. Natl. Acad. Sci. U.S.A.

270

234

View full text Add to dashboard Cite

A lack of automated, quantitative, and accurate assessment of social behaviors in mammalian animal models has limited progress toward understanding mechanisms underlying social interactions and their disorders such as autism. Here we present a new integrated hardware and software system that combines video tracking, depth sensing, and machine learning for automatic detection and quantification of social behaviors involving close and dynamic interactions between two mice of different coat colors in their home cage. We designed a hardware setup that integrates traditional video cameras with a depth camera, developed computer vision tools to extract the body "pose" of individual animals in a social context, and used a supervised learning algorithm to classify several well-described social behaviors. We validated the robustness of the automated classifiers in various experimental settings and used them to examine how genetic background, such as that of Black and Tan Brachyury (BTBR) mice (a previously reported autism model), influences social behavior. Our integrated approach allows for rapid, automated measurement of social behaviors across diverse experimental designs and also affords the ability to develop new, objective behavioral metrics.

show abstract

“…In supervised learning, the human expert identifies and categorizes patterns in the data by informing the software of categorical behavior annotations [8,15,71,73] (Movie S11, Movie S12, and Movie S14 in the supplementary material online). For example, 'wing grooming' could represent when a fly rubs one or both metathoracic legs over the top or the underside of the wing(s) [73].…”

Section: Automated Behavioral Analysismentioning

confidence: 99%

“…For example, 'wing grooming' could represent when a fly rubs one or both metathoracic legs over the top or the underside of the wing(s) [73]. Categorical annotations like these simultaneously take into account many different features of the trajectory and pose data, and result in categories that are generally easily interpretable by biologists and often have a clear physiological or ecological significance.…”

Section: Automated Behavioral Analysismentioning

confidence: 99%

Automated image-based tracking and its application in ecology

Dell

Bender²,

Branson

et al. 2014

Trends in Ecology & Evolution

Self Cite

445

484

View full text Add to dashboard Cite

The behavior of individuals determines the strength and outcome of ecological interactions, which drive population, community, and ecosystem organization. Bio-logging, such as telemetry and animal-borne imaging, provides essential individual viewpoints, tracks, and life histories, but requires capture of individuals and is often impractical to scale. Recent developments in automated image-based tracking offers opportunities to remotely quantify and understand individual behavior at scales and resolutions not previously possible, providing an essential supplement to other tracking methodologies in ecology. Automated image-based tracking should continue to advance the field of ecology by enabling better understanding of the linkages between individual and higher-level ecological processes, via high-throughput quantitative analysis of complex ecological patterns and processes across scales, including analysis of environmental drivers. Measuring behaviorIndividual behavior (see Glossary) underlies almost all aspects of ecology [1][2][3][4][5]. Accurate and highly resolved behavioral data are therefore critical for obtaining a mechanistic and predictive understanding of ecological systems [5]. Historically, direct observation by trained biologists was used to quantify behavior [6,7]. However, the extent and resolution to which direct observations can be made is highly constrained [8] and the number of individuals that can be observed simultaneously is small. In addition, an exact record of events is not preserved, only the biologist's subjective account of them.Recent technological advances in tracking now make it possible to collect large amounts of highly precise and accurate behavioral data. For many organisms equipment can be attached that provide information about the Glossary Background subtraction: a method used by software to compare the current video frame with a stored picture of the background; any pixel of the current frame that is significantly different from the corresponding pixel in the background is likely to be associated with the body of an animal. Useful in situations where the background is unchanging, for example, when the surface of the background is rigid and lighting does not change. Behavior: the actions of individuals, often in response to stimuli. Behavior can involve movement of the individual's body through space, such as walking or chasing, or can occur while the animal is stationary, such as grooming or eating. Bio-logging: attachment or implantation of equipment to organisms to provide information about their identity, location, behavior, or physiology (e.g., global positioning systems, accelerometers, video cameras, telemetry tags). Ecological interaction: any interaction between an organism and its environment, or between two organisms (i.e., including interactions between conspecifics). Fingerprinting: a method used to identify unmarked individuals using natural variation in their physical and/or behavioral appearance. The method works by transforming the images of each individual ...

show abstract

JAABA: interactive machine learning for automatic annotation of animal behavior

Cited by 528 publications

References 13 publications

Detecting Social Actions of Fruit Flies

Detecting Social Actions of Fruit Flies

Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning

Automated image-based tracking and its application in ecology

Contact Info

Product

Resources

About