ZooBuilder: 2D and 3D Pose Estimation for Quadrupeds Using Synthetic Data

Fangbemi, Abassin Sourou; Lu, Yi; Xu, Mao Yuan; Luo, Xiao Wu; Rolland, Alexis; Raïssi, Chedy

doi:10.48550/arxiv.2009.05389

Cited by 2 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, the combination of depth passes and camera intrinsics and extrinsics can in principle be used to train networks to infer 3D locations directly from a single 2D image[36, 40, 69]; informing posture variation within replicAnt with 3D kinematics data from live animals[30, 50], may yield networks which can infer the location of occluded key points with reasonable accuracy; and further annotation, for example on minute species differences or body size, can readily be appended. Recent advancements in styleand domain transfer could be combined with data produced by replicAnt to produce even stronger generalist[30, 31, 67], or application specific networks[48–50]. The domain-gap may be narrowed further by introduction of novel pre-trained networks, such as Segment Anything[70] and DINO(v2)[33], as feature-extraction backbones.…”

Section: Discussionmentioning

confidence: 99%

“…By placing 3D models in simulated environments, variable and annotated datasets can be generated at scale, and at a fraction of the cost and time required for hand-annotation of real images [39,40,42,44]. The use of synthetic data is particularly attractive where annotated real datasets are practically absent or only of insufficient size, as is the case for almost all non-human animal studies [22,[30][31][32][45][46][47][48][49][50][51]. However, for all its conceptual attractiveness, using synthetic data is not without problems: the simulation must bridge the "simulationreality gap", i. e. the synthetic image must be comparable in appearance to real images; as before, the key challenge remains that the training data must represent a superset of the inputs received at inference time [22,27,[48][49][50].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

replicAnt: A pipeline for generating annotated images of animals in complex environments using Unreal Engine

Plum

Bulla²,

Beck

et al. 2023

Preprint

View full text Add to dashboard Cite

Deep learning-based computer vision methods are transforming animal behavioural research. Transfer learning has enabled work in non-model species, but still requires hand-annotation of example footage, and is only performant in well-defined conditions. To overcome these limitations, we created replicAnt, a configurable pipeline implemented in Unreal Engine 5 and Python, designed to generate large and variable training datasets on consumer-grade hardware instead. replicAnt places 3D animal models into complex, procedurally generated environments, from which automatically annotated images can be exported. We demonstrate that synthetic data generated with replicAnt can significantly reduce the hand-annotation required to achieve benchmark performance in common applications such as animal detection, tracking, pose-estimation, and semantic segmentation; and that it increases the subject-specificity and domain-invariance of the trained networks, so conferring robustness. In some applications, replicAnt may even remove the need for hand-annotation altogether. It thus represents a significant step towards porting deep learning-based computer vision tools to the field.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

replicAnt: A pipeline for generating annotated images of animals in complex environments using Unreal Engine

Plum

Bulla²,

Beck

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…A. S. Fangbemi et al [22] deliberated the 2D and 3D Pose Estimation for Quadrupeds Using Synthetic Data. This study uses keyframe animations to present a new method for creating synthetic training data for 3D and 2D animal posture estimation.…”

Section: Related Studymentioning

confidence: 99%

Research on 2D Animation Simulation Based on Artificial Intelligence and Biomechanical Modeling

Fangming Dai,

2024

EAI Endorsed Trans Perv Health Tech

View full text Add to dashboard Cite

Animation techniques have been completely transformed by the union of Artificial Intelligence (AI) and biomechanical modeling, particularly in 2D animation. This study looks at a combination of AI and biomechanics to address the challenges of simulating 2D animation. Current approaches in 2D animation often struggle to achieve lifelike and fluid movements, especially when representing complex motion or interaction. These traditional techniques rely on manual keyframing or physics simulation, which may be time-consuming and do not provide the rich detail needed for realism in animations. To meet these aspects, this study suggested 2D animation using Artificial Intelligence with Biomechanical Modeling (2D-AI-BM). Our approach thus harnesses Deep Neural Network (DNN) for moving forecasts and improvement using biopsychological principles to help us imitate natural human actions better. In addition to character animation, it could apply to interactive storytelling and educational simulations. As a result, animators get more control over motion generation while drastically reducing the necessity for manual intervention through this fusion of AI and biomechanics, which smoothens the production pipeline for animations. This paper considers several important metrics to evaluate the proposed approach’s effectiveness, including user satisfaction, computational efficiency, motion smoothness and realism. Comparative studies with classical animation methods showed that the method generates realistic movements on 2D characters while saving time during production. The numerical findings exemplify that the recommended 2D-AI-BM model improves an accuracy rate of 97.4%, computational efficiency ratio of 96.3%, motion control ratio of 95.4%, pose detection ratio of 94.8% and scalability ratio of 93.2% compared to other popular techniques.

show abstract

ZooBuilder: 2D and 3D Pose Estimation for Quadrupeds Using Synthetic Data

Cited by 2 publications

References 0 publications

replicAnt: A pipeline for generating annotated images of animals in complex environments using Unreal Engine

replicAnt: A pipeline for generating annotated images of animals in complex environments using Unreal Engine

Research on 2D Animation Simulation Based on Artificial Intelligence and Biomechanical Modeling

Contact Info

Product

Resources

About