TRex, a fast multi-animal tracking system with markerless identification, and 2D estimation of posture and visual fields

Abstract

Automated visual tracking of animals is rapidly becoming an indispensable tool for the study of behavior. It offers a quantitative methodology by which organisms' sensing and decision-making can be studied in a wide range of ecological contexts. Despite this, existing solutions tend to be challenging to deploy in practice, especially when considering long and/or high-resolution video-streams. Here, we present TRex, a fast and easy-to-use solution for tracking a large number of individuals simultaneously using background-subtraction with real-time (60Hz) tracking performance for up to approximately 256 individuals and estimates 2D visual-fields, outlines, and head/rear of bilateral animals, both in open and closed-loop contexts. Additionally, TRex offers highly-accurate, deep-learning-based visual identification of up to approximately 100 unmarked individuals, where it is between 2.5-46.7 times faster, and requires 2-10 times less memory, than comparable software (with relative performance increasing for more organisms/longer videos) and provides interactive data-exploration within an intuitive, platform-independent graphical user-interface.

Ethoflow: Computer Vision and Artificial Intelligence-Based Software for Automatic Behavior Analysis

“…Multivariate assessment of behavioral traits can bring important insights into animals’ ecological aspects, for instance, in studies of toxicological assessments and animal behavior [ 41 , 42 , 43 ]. Some modern software programs that use deep learning to evaluate behaviors demand powerful machines with GPU [ 17 , 18 , 44 ], which makes the analysis of laboratory routines in ordinary computers difficult. In the Ethoflow algorithm, the AI processing is separate from the other modules, enabling kinematic measurements on an ordinary computer and assessing more complex behavior using a GPU.…”

“…This software exhibited advantages in processing speed and robustness compared to the software IdTracker [ 14 ] and the ToxTrac [ 15 ]. Some other tracking software exhibit outstanding performance using deep learning algorithms [ 16 ], including the idtracker.ai [ 17 ] and the TRex [ 18 ]. These two software also apply CNNs to track many animals simultaneously with high accuracy in maintaining individuals’ identities.…”

“…The available tracking software measure large collectively animal groups with high accuracy, especially those using deep learning. However, these tracking software operate by background subtraction or thresholding [ 13 , 14 , 15 , 17 , 18 ]. These approaches require video recordings of homogeneous environments and are not applicable in the field.…”

Estimating the effect of tracking tag weight on insect movement using video analysis: A case study with a flightless orthopteran

“…We, therefore, developed our own application that enables the identity tracking of insects [48]. After we developed and published our algorithm, three other interesting approaches were published, i.e., ToxTrac [49], GRAPHITE [35], and TRex [50]. ToxTrac is fast and simple, but we found that it generated new IDs if occlusion occurs (unpublished observations), which was not acceptable for our experiment.…”

New Open-Source Tools: Using Bonsai for Behavioral Tracking and Closed-Loop Experiments

“…Furthermore, first-order features can often be combined to yield other measures of interest. For example, tracking head position can be used to infer what portion of the visual field is accessible to the animal at each moment, thus allowing the researcher to precisely determine which visual information is accessible to the animal, despite freely moving conditions (Walter and Couzin, 2020).…”