Action Detection for Wildlife Monitoring with Camera Traps Based on Segmentation with Filtering of Tracklets (SWIFT) and Mask-Guided Action Recognition (MAROON)

Abstract

Behavioral analysis of animals in the wild plays an important role for ecological research and conservation and has been mostly performed by researchers. We introduce an action detection approach that automates this process by detecting animals and performing action recognition on the detected animals in camera trap videos. Our action detection approach is based on SWIFT (segmentation with filtering of tracklets), which we have already shown to successfully detect and track animals in wildlife videos, and MAROON (mask-guided action recognition), an action recognition network that we are introducing here. The basic ideas of MAROON are the exploitation of the instance masks detected by SWIFT and a triple-stream network. The instance masks enable more accurate action recognition, especially if multiple animals appear in a video at the same time. The triple-stream approach extracts features for the motion and appearance of the animal. We evaluate the quality of our action recognition on two self-generated datasets, from an animal enclosure and from the wild. These datasets contain videos of red deer, fallow deer and roe deer, recorded both during the day and night. MAROON improves the action recognition accuracy compared to other state-of-the-art approaches by an average of 10 percentage points on all analyzed datasets and achieves an accuracy of 69.16% on the Rolandseck Daylight dataset, in which 11 different action classes occur. Our action detection system makes it possible todrasticallyreduce the manual work of ecologists and at the same time gain new insights through standardized results.

The Potential of Diffusion-Based Near-Infrared Image Colorization

“…The integration of camera trapping with artificial intelligence (AI), particularly leveraging deep learning methodologies, represents a significant advancement in wildlife research and conservation [ 3 , 7 , 11 , 12 ]. Nevertheless, numerous deep learning models are optimized for and perform better with colored images, akin to human perception [ 10 ].…”

“… These camera-trap-based documentation records allow for estimations of animal populations [ 1 ] and movements of animals and herds [ 2 ]. Camera traps can record animal behavior [ 3 ]. Using invisible infrared flashlights, camera traps work non-invasive and, therefore, have no disturbing effects on animal behavior.…”

“…Additionally, in recent years the combination of camera trapping and of artificial intelligence (AI), especially of deep learning approaches, has emerged as a breakthrough in the field of wildlife research and conservation [ 3 , 7 , 11 , 12 ]. However, many deep learning approaches are trained for and can benefit from colored images [ 13 ] like humans do [ 10 ].…”

Zero-shot animal behavior classification with vision-language foundation models

“…However, one might want to study more informative, behavior-specific patterns, for instance to understand when are animals foraging, resting or moving. In contrast with species identification, classification of animals' behavior in camera-trap images or video is still in its infancy (Norouzzadeh et al 2018;Liu et al 2023;Schindler and Steinhage 2021;Schindler, Steinhage, et al 2024).…”