A Gated Fusion Network for Dynamic Saliency Prediction

Kocak, Aysun; Erdem, Erkut; Erdem, Aykut

doi:10.1109/tcds.2021.3094974

Cited by 6 publications

(3 citation statements)

References 80 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One network uses red, green, and blue (RGB) images, the other network uses optical flows as input, and the spatiotemporal features extracted from each are fused and used for prediction. Zhang et al [25], Wu et al [26], and Kocak et al [27] studied how to fuse spatiotemporal information to improve the performance of a video saliency prediction model consisting of two streams. Fu et al [28] proposed UVANet which merges two streams through transfer learning.…”

Section: Introductionmentioning

confidence: 99%

OFF-ViNet: Optical Flow-Based Feature Warping ViNet for Video Saliency Prediction Considering Future Prediction

Ikenoya,

Tashiro,

Ohashi

2024

IEEE Access

View full text Add to dashboard Cite

Active studies have been conducted on video saliency prediction, which predicts human visual attention toward videos. Most deep learning-based video saliency prediction models implicitly learn features that contribute to video saliency prediction, greatly improving accuracy. This study proposes a model called optical flow-based feature warping ViNet (OFF-ViNet). This model explicitly adds a Warping module, which is a mechanism that considers future predictions based on object motion in addition to implicitly learned features. The Warping module spatially warps the hierarchical features extracted by the 3D convolutional backbone based on the optical flow to obtain a feature representation that predicts the future. Compared with exisiting models, OFF-ViNet achieves better and competitive accuracy with state-of-the-art models on video saliency prediction datasets, particularly on UCF-Sports, which contains several videos with moving objects.

show abstract

Section: Introductionmentioning

confidence: 99%

OFF-ViNet: Optical Flow-Based Feature Warping ViNet for Video Saliency Prediction Considering Future Prediction

Ikenoya,

Tashiro,

Ohashi

2024

IEEE Access

View full text Add to dashboard Cite

show abstract

Section: Gated Fusion Mechanismmentioning

confidence: 99%

“…The gated fusion mechanism [64,65] is usually utilized to fuse the features of multimodal data sources. Inspired by Kocak et.al [64], this mechanism would learn some gate values, which could represent the contribution of each data source. Then the features would be weighted accordingly for the generation of the final output, i.e., the features from the Classification and Segmentation branches would be fused in a way of weighted sum, to produce the final output S out .…”

Section: Gated Fusion Mechanismmentioning

confidence: 99%

Weakly supervised learning on road area extraction via classification labels

Liu¹

View full text Add to dashboard Cite

I am especially thankful to Asst Prof Long Cheng, my supervisor, mentor, and friend, who deserves much of the credit for this thesis. When I was faced with a gap year, he went out of his way to help me and gave me a chance to pursue the degree. Prof. Long sparked my initial interest in research and taught me how to conduct and communicate research. Prof. Long's guidance, wisdom, and support has made it possible for me to achieve my potential.I am so thankful that Xu Qianxiong helps me a lot during the experiment design and points out some room for improvement in my research. My thanks also go to

show abstract