Hierarchical Bayesian LSTM for Head Trajectory Prediction on Omnidirectional Images

Yang, Li; Xu, Mai; Guo, Yichen; Deng, Xin; Gao, Fangyuan; Guan, Zhenyu

doi:10.1109/tpami.2021.3117019

Cited by 13 publications

(8 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Considering uncertainty in prediction is often approached with Bayesian neural networks [33,45,70]. However, these approaches are computationally intensive and do not allow to capture mode diversity in the data [20].…”

Section: Discussionmentioning

confidence: 99%

“…Kan et al [33] considered Bayesian neural networks (BNN) to output the probability distribution of future throughput, given the network's historical throughput. In a very recent work, Yang et al [70] considered predicting multiple head trajectories but only for 360°images, not videos as we do. They consider head trajectory as a succession of fixations and saccades, and intend to learn to capture the uncertainty of head trajectories across different subjects.…”

Section: Considering Prediction Uncertaintymentioning

confidence: 99%

“…We therefore cope not only with inter-user variability, but also with intrinsic uncertainty of the data in how past is correlated with future motion, data uncertainty often referred to as aleatoric uncertainty. Also, BNN are computationally-heavy (the approach from Yang et al [70] is not real-time) and fit accurately but to just one mode in the data [20]. In this article, we consider a lightweight approach to multiple trajectory prediction, able to predict multiple modes for the future trajectory.…”

Section: Considering Prediction Uncertaintymentioning

confidence: 99%

“…However, the performance of existing prediction models is limited by a main characteristic of the head motion data: its intrinsic uncertainty. Very few models have considered this characteristic so far [17,29,70], but only heuristically for 360°v ideos. We illustrate this uncertainty in Fig.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Deep variational learning for multiple trajectory prediction of 360° head movements

Guimard

Sassatelli

Marchetti

et al. 2022

Proceedings of the 13th ACM Multimedia Systems Conference

View full text Add to dashboard Cite

Prediction of head movements in immersive media is key to design efficient streaming systems able to focus the bandwidth budget on visible areas of the content. Numerous proposals have therefore been made in the recent years to predict 360°images and videos. However, the performance of these models is limited by a main characteristic of the head motion data: its intrinsic uncertainty. In this article, we present an approach to generate multiple plausible futures of head motion in 360°videos, given a common past trajectory. Our method provides likelihood estimates of every predicted trajectory, enabling direct integration in streaming optimization. To the best of our knowledge, this is the first work that considers the problem of multiple head motion prediction for 360°video streaming. We first quantify this uncertainty from the data. We then introduce our discrete variational multiple sequence (DVMS) learning framework, which builds on deep latent variable models. We design a training procedure to obtain a flexible and lightweight stochastic prediction model compatible with sequence-to-sequence recurrent neural architectures. Experimental results on 3 different datasets show that our method DVMS outperforms competitors adapted from the selfdriving domain by up to 37% on prediction horizons up to 5 sec., at lower computational and memory costs. Finally, we design a method to estimate the respective likelihoods of the multiple predicted trajectories, by exploiting the stationarity of the distribution of the prediction error over the latent space. Experimental results on 3 datasets show the quality of these estimates, and how they depend on the video category. CCS CONCEPTS• Human-centered computing → Virtual reality; • Information systems → Multimedia streaming; • Computing methodologies → Neural networks.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Considering Prediction Uncertaintymentioning

confidence: 99%

Section: Considering Prediction Uncertaintymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Deep variational learning for multiple trajectory prediction of 360° head movements

Guimard

Sassatelli

Marchetti

et al. 2022

Proceedings of the 13th ACM Multimedia Systems Conference

View full text Add to dashboard Cite

show abstract

“…However, with a background in materials science, the ML methods are so outdated that they cannot be satisfied with many application scenarios. For example, most ANN models are based on feedforward neural networks, while different types of neural networks such as time series networks [ 104 , 105 , 106 ], recurrent neural networks [ 107 , 108 , 109 ], and adversarial networks [ 110 , 111 , 112 ] have appeared in the field of computer science. Later, the use of various new neural networks mixed in materials science will become a future trend.…”

Section: Outlook and Conclusionmentioning

confidence: 99%

A Review of Performance Prediction Based on Machine Learning in Materials Science

Liu

Huang

et al. 2022

Nanomaterials

View full text Add to dashboard Cite

With increasing demand in many areas, materials are constantly evolving. However, they still have numerous practical constraints. The rational design and discovery of new materials can create a huge technological and social impact. However, such rational design and discovery require a holistic, multi-stage design process, including the design of the material composition, material structure, material properties as well as process design and engineering. Such a complex exploration using traditional scientific methods is not only blind but also a huge waste of time and resources. Machine learning (ML), which is used across data to find correlations in material properties and understand the chemical properties of materials, is being considered a new way to explore the materials field. This paper reviews some of the major recent advances and applications of ML in the field of properties prediction of materials and discusses the key challenges and opportunities in this cross-cutting area.

show abstract

DNA Tetrahedron Delivering miR‐21‐5p Promotes Senescent Bone Defects Repair through Synergistic Regulation of Osteogenesis and Angiogenesis

Qi,

Hong,

Zhao

et al. 2024

Adv Healthcare Materials

View full text Add to dashboard Cite

Compromised osteogenesis and angiogenesis is the character of stem cell senescence, which brought difficulties for bone defects repairing in senescent microenvironment. As the most abundant bone‐related miRNA, miRNA‐21‐5p plays a crucial role in inducing osteogenic and angiogenic differentiation. However, highly efficient miR‐21‐5p delivery still confronts challenges including poor cellular uptake and easy degradation. Herein, TDN‐miR‐21‐5p nanocomplex is constructed based on DNA tetrahedral (TDN) and has great potential in promoting osteogenesis and alleviating senescence of senescent bone marrow stem cells (O‐BMSCs), simultaneously enhancing angiogenic capacity of senescent endothelial progenitor cells (O‐EPCs). Of note, the activation of AKT and Erk signaling pathway may direct regulatory mechanism of TDN‐miR‐21‐5p mediated osteogenesis and senescence of O‐BMSCs. Also, TDN‐miR‐21‐5p can indirectly mediate osteogenesis and senescence of O‐BMSCs through pro‐angiogenic growth factors secreted from O‐EPCs. In addition, gelatin methacryloyl (GelMA) hydrogels are mixed with TDN and TDN‐miR‐21‐5p to fabricate delivery scaffolds. TDN‐miR‐21‐5p@GelMA scaffold exhibits greater bone repair with increased expression of osteogenic‐ and angiogenic‐related markers in senescent critical‐size cranial defects in vivo. Collectively, TDN‐miR‐21‐5p can alleviate senescence and induce osteogenesis and angiogenesis in senescent microenvironment, which provides a novel candidate strategy for senescent bone repair and widen clinical application of TDNs‐based gene therapy.

show abstract

Hierarchical Bayesian LSTM for Head Trajectory Prediction on Omnidirectional Images

Cited by 13 publications

References 53 publications

Deep variational learning for multiple trajectory prediction of 360° head movements

Deep variational learning for multiple trajectory prediction of 360° head movements

A Review of Performance Prediction Based on Machine Learning in Materials Science

DNA Tetrahedron Delivering miR‐21‐5p Promotes Senescent Bone Defects Repair through Synergistic Regulation of Osteogenesis and Angiogenesis

Contact Info

Product

Resources

About