Pose Representations for Deep Skeletal Animation

Nefeli, Andreou,; Aristidou, Andreas; Chrysanthou, Yiorgos

doi:10.1111/cgf.14632

Cited by 5 publications

(5 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These hierarchies allow for efficient manipulation and animation of the entire skeleton through techniques such as keyframe animation, forward and inverse kinematics and motion capture. Rotations in these representations are typically represented using Euler Angles, Quaternions [PGA18], Rotation Matrices (or 6D representations) [ZBL * 19] or variations such as Dual Quaternions [AAC22]. The storage and retrieval of this type of data is usually achieved using suitable motion capture file formats and protocols.…”

Section: Motion Representation and Storagementioning

confidence: 99%

Virtual Instrument Performances (VIP): A Comprehensive Review

Kyriakou,

de la Campa Crespo,

Panayiotou

et al. 2024

Computer Graphics Forum

Self Cite

View full text Add to dashboard Cite

Driven by recent advancements in Extended Reality (XR), the hype around the Metaverse, and real‐time computer graphics, the transformation of the performing arts, particularly in digitizing and visualizing musical experiences, is an ever‐evolving landscape. This transformation offers significant potential in promoting inclusivity, fostering creativity, and enabling live performances in diverse settings. However, despite its immense potential, the field of Virtual Instrument Performances (VIP) has remained relatively unexplored due to numerous challenges. These challenges arise from the complex and multi‐modal nature of musical instrument performances, the need for high precision motion capture under occlusions including the intricate interactions between a musician's body and fingers with instruments, the precise synchronization and seamless integration of various sensory modalities, accommodating variations in musicians' playing styles, facial expressions, and addressing instrument‐specific nuances. This comprehensive survey delves into the intersection of technology, innovation, and artistic expression in the domain of virtual instrument performances. It explores musical performance multi‐modal databases and investigates a wide range of data acquisition methods, encompassing diverse motion capture techniques, facial expression recording, and various approaches for capturing audio and MIDI data (Musical Instrument Digital Interface). The survey also explores Music Information Retrieval (MIR) tasks, with a particular emphasis on the Musical Performance Analysis (MPA) field, and offers an overview of various works in the realm of Musical Instrument Performance Synthesis (MIPS), encompassing recent advancements in generative models. The ultimate aim of this survey is to unveil the technological limitations, initiate a dialogue about the current challenges, and propose promising avenues for future research at the intersection of technology and the arts.

show abstract

Section: Motion Representation and Storagementioning

confidence: 99%

Virtual Instrument Performances (VIP): A Comprehensive Review

Kyriakou,

de la Campa Crespo,

Panayiotou

et al. 2024

Computer Graphics Forum

Self Cite

View full text Add to dashboard Cite

show abstract

“…First, we retrieve the positions and rotations from six sensors placed on the head, hands, feet and pelvis (the root in our case) of the user. Then, these are transformed into a root-centered dual quaternion-based pose representation [Andreou et al 2022], which allows the network to implicitly understand the structure of the skeleton and synthesize accurate poses. A convolutional-based autoencoder extracts the main features from the sensors and reconstructs the user poses for a set of contiguous frames.…”

Section: Overviewmentioning

confidence: 99%

“…For the dynamic part, Q, we represent the local rotations and translations using unit dual quaternions, as presented by Andreou et al [2022]. Dual quaternions provide a unified and compact representation that encodes both rotational and translation information in orthogonal quaternions, allowing the network to understand human motion better.…”

Section: Input and Pose Representationmentioning

confidence: 99%

“…In our method, we use dual quaternions, as proposed by Andreou et al [2022], as our pose representation because they offer a unified and concise representation that includes both rotation and translation information. Through our experimentation, we have discovered that utilizing dual quaternions results in superior reconstruction of poses and continuity compared to other commonly used pose representations such as quaternions or ortho6D [Zhou et al 2019].…”

Section: Pose Representationmentioning

confidence: 99%

“…This section provides the fundamental concepts essential to understanding the proposed method. Specifically, we introduce dual quaternions, which serve as the pose representation utilized in Section 5.1 as presented by Andreou et al [2022], as well as the skeleton-aware operations introduced by Aberman et al [2020] which we use as a component of our network architecture.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

Ponton,

Yun,

Aristidou

et al. 2023

ACM Trans. Graph.

Self Cite

View full text Add to dashboard Cite

Accurate and reliable human motion reconstruction is crucial for creating natural interactions of full-body avatars in Virtual Reality (VR) and entertainment applications. As the Metaverse and social applications gain popularity, users are seeking cost-effective solutions to create full-body animations that are comparable in quality to those produced by commercial motion capture systems. In order to provide affordable solutions though, it is important to minimize the number of sensors attached to the subject’s body. Unfortunately, reconstructing the full-body pose from sparse data is a heavily under-determined problem. Some studies that use IMU sensors face challenges in reconstructing the pose due to positional drift and ambiguity of the poses. In recent years, some mainstream VR systems have released 6-degree-of-freedom (6-DoF) tracking devices providing positional and rotational information. Nevertheless, most solutions for reconstructing full-body poses rely on traditional inverse kinematics (IK) solutions, which often produce non-continuous and unnatural poses. In this paper, we introduce SparsePoser, a novel deep learning-based solution for reconstructing a full-body pose from a reduced set of six tracking devices. Our system incorporates a convolutional-based autoencoder that synthesizes high-quality continuous human poses by learning the human motion manifold from motion capture data. Then, we employ a learned IK component, made of multiple lightweight feed-forward neural networks, to adjust the hands and feet towards the corresponding trackers. We extensively evaluate our method on publicly available motion capture datasets and with real-time live demos. We show that our method outperforms state-of-the-art techniques using IMU sensors or 6-DoF tracking devices, and can be used for users with different body dimensions and proportions.

show abstract

Deep fusible skinning of animation sequences

Moutafidou,

Toulatzis,

Fudos

2023

Vis Comput

View full text Add to dashboard Cite

Animation compression is a key process in replicating and streaming animated 3D models. Linear Blend Skinning (LBS) facilitates the compression of an animated sequence while maintaining the capability of real-time streaming by deriving vertex to proxy bone assignments and per frame bone transformations. We introduce a innovative deep learning approach that learns how to assign vertices to proxy bones with persistent labeling. This is accomplished by learning how to correlate vertex trajectories to bones of fully rigged animated 3D models. Our method uses these pretrained networks on dynamic characteristics (vertex trajectories) of an unseen animation sequence (a sequence of meshes without skeleton or rigging information) to derive an LBS scheme that outperforms most previous competent approaches by offering better approximation of the original animation sequence with fewer bones, therefore offering better compression and smaller bandwidth requirements for streaming. This is substantiated by a thorough comparative performance evaluation using several error metrics, and compression/bandwidth measurements. In this paper, we have also introduced a persistent bone labeling scheme that (i) improves the efficiency of our method in terms of lower error values and better visual outcome and (ii) facilitates the fusion of two (or more) LBS schemes by an innovative algorithm that combines two arbitrary LBS schemes. To demonstrate the usefulness and potential of this fusion process, we have combined the outcome of our deep skinning method with that of Rignet—which is a state-of-the-art method that performs rigging on static meshes—with impressive results.

show abstract

Pose Representations for Deep Skeletal Animation

Cited by 5 publications

References 42 publications

Virtual Instrument Performances (VIP): A Comprehensive Review

Virtual Instrument Performances (VIP): A Comprehensive Review

SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

Deep fusible skinning of animation sequences

Contact Info

Product

Resources

About