replicAnt: A pipeline for generating annotated images of animals in complex environments using Unreal Engine

Abstract

Deep learning-based computer vision methods are transforming animal behavioural research. Transfer learning has enabled work in non-model species, but still requires hand-annotation of example footage, and is only performant in well-defined conditions. To overcome these limitations, we created replicAnt, a configurable pipeline implemented in Unreal Engine 5 and Python, designed to generate large and variable training datasets on consumer-grade hardware instead. replicAnt places 3D animal models into complex, procedurally generated environments, from which automatically annotated images can be exported. We demonstrate that synthetic data generated with replicAnt can significantly reduce the hand-annotation required to achieve benchmark performance in common applications such as animal detection, tracking, pose-estimation, and semantic segmentation; and that it increases the subject-specificity and domain-invariance of the trained networks, so conferring robustness. In some applications, replicAnt may even remove the need for hand-annotation altogether. It thus represents a significant step towards porting deep learning-based computer vision tools to the field.

WOLO: Wilson Only Looks Once – Estimating ant body mass from reference-free images using deep convolutional neural networks

“…This effect was particularly evident for recordings from cluttered environments (Fig. 10, B,C,H,I), and mirrors similar results in earlier work, which suggested that synthetic data can help to embed a subject-specific understanding into the networks [45]. In the context of body mass inference, a key strength of synthetic data is that it can be generated such that size-related differences in cropped image occupancy or compression artefacts are entirely avoided, so preventing networks from learning to infer size from these artefactual features, which would impede generalisation.…”

“…To implement augmentation on a large scale, we produced thousands of computer-generate images that possesses far greater variability in appearance than the original training data. Synthetic datasets were generated with replicAnt , a computational data generation pipeline implemented in Unreal Engine 5 and Python [45]. replicAnt takes textured and rigged 3D animal models as an input, and places simulated populations of these models into complex, procedurally generated environments.…”

“…Unreal Engine 5 and Python [45]. replicAnt takes textured and rigged 3D animal models as an input, and places simulated populations of these models into complex, procedurally generated environments.…”

“…Within each of the 20 size classes, a randomised scale variation of 10% was applied, so that adjacent weight classes did not overlap in absolute scale. Synth-standard used the "plants" environment provided with replicAnt to facilitate a complex scene generation with plant asset scatterers [45]. The resulting image samples were highly cluttered, and often included individuals spawned across multiple layers with respect to the camera plane.…”

“…Fig.7To increase network robustness and generalisability, real datasets were augmented with two synthetic datasets, computer-generated with replicAnt[45]. (A) 20 "digital twins" of leaf-cutter ant workers across the mass-range[1,50] mg were created with scAnt[44], an open-source photogrammetry platform (see Supplementary Table1).…”