Deep Spatial-Temporal Neural Network for Dense Non-Rigid Structure from Motion

Abstract

Dense non-rigid structure from motion (NRSfM) has long been a challenge in computer vision because of the vast number of feature points. As neural networks develop rapidly, a novel solution is emerging. However, existing methods ignore the significance of spatial–temporal data and the strong capacity of neural networks for learning. This study proposes a deep spatial–temporal NRSfM framework (DST-NRSfM) and introduces a weighted spatial constraint to further optimize the 3D reconstruction results. Layer normalization layers are applied in dense NRSfM tasks to stop gradient disappearance and hasten neural network convergence. Our DST-NRSfM framework outperforms both classical approaches and recent advancements. It achieves state-of-the-art performance across commonly used synthetic and real benchmark datasets.

Temporal-Aware Neural Network for Dense Non-Rigid Structure from Motion

“…In the evaluation of the Synthetic Face dataset, a comparison was conducted between the 3D reconstruction framework employed and classical traditional methods such as VA [8] and CMDR [39]. Additionally, an assessment was made of newer traditional methods, including GM [9], JM [19], SMSR [21], PPTA [40], and EM-FEM [7], along with neural network methods like N-NRSfM [6], RONN [41], NTP [38], and DST-NRSFM [42]. Furthermore, comparisons were carried out with other relevant methods.…”

“…VA [8] CMDR [39] PPTA [40] JM [19] SMSR [21] GM [9] Traj Actor Mocap: The Actor Motion Capture dataset comprises 100 frames, encompassing a total of 36,349 points. To gauge the effectiveness of the method on this dataset, a comparative evaluation was conducted against FML [43], SMSR [21], CMDR [39], RONN [41], N-NRSFM [6], and DST-NRSFM [42]. The outcomes, as shown in Table 2, reveal the superior performance and accuracy achieved by the proposed method.…”

“…Expressions: The Expression dataset, characterized by its relative sparsity, consists of 384 frames and 997 points. In this study, a comparative analysis of this dataset was conducted involving several methods, namely, CSF2 [44], KSTA [17], GMLI [45], N-NRSFM [6], RONN [41], and DST-NRSFM [42]. The conclusive experimental results are presented in Table 3.…”

“…FML [43] SMSR [21] CMDR [39] RONN [41] N-NRSFM [6] DST-NRSFM [ Pants: The Pants dataset exhibits a relatively sparse nature, comprising a total of 291 frames and 2859 points. In this study, an assessment of various methods on this dataset was conducted, encompassing KSTA [17], BMM [20], PPTA [40], and DST-NRSFM [42]. The outcomes of these comparative experiments are detailed in Table 4.Through a comprehensive analysis and comparison of the experimental results presented in both Tables 3 and 4, it becomes evident that, despite the inherent sparsity of both the Expressions and Pants datasets, significant improvements in accuracy and robustness are achieved using the proposed method.…”

“…KSTA [17] BMM [20] PPTA [40] DST-NRSFM [42] Ours pants 0.220 0.183 0.203 0.148 0.087 Kinect Sequence: The Kinect sequence dataset consists of two subsets: the Paper dataset and the T-shirt dataset, encompassing 193 frames with 58,000 points and 313 frames with 77,000 points, respectively. In this study, a comparison was made with MP [46], DSTA [47], GM [9], JM [19], N-NRSFM [6], and DST-NRSFM [42], and the detailed results can be found in Table 5.…”