Object Relational Graph With Teacher-Recommended Learning for Video Captioning

Zhang, Ziqi; Shi, Yaya; Yuan, Chunfeng; Li, Bing; Wang, Peijin; Hu, Weiming; Zha, Zheng-Jun

doi:10.1109/cvpr42600.2020.01329

Cited by 226 publications

(99 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note that many of them were published after our initial submission of the present work in the end of May in 2019. Specifically, compared with ORG-TRL (Zhang et al, 2020 ), the previous state-of-the-art model on this dataset, BLEU-4, CIDEr, METEOR, and ROUGE-L are improved relatively by 14.9, 15.2, 7.1, and 4.2%, respectively. Our model has the highest overall score as defined in (16).…”

Section: Methodsmentioning

confidence: 90%

“…Table 1 displays the performance of several models on YouTube2Text. We compare our model with existing methods, including LSTM-E (Pan et al, 2016 ), h-RNN (Yu et al, 2016 ), aLSTMs (Gao et al, 2017 ), SCN (Gan et al, 2017 ), MTVC (Pasunuru and Bansal, 2017a ), ECO (Zolfaghari et al, 2018 ), SibNet (Liu et al, 2018 ), POS (Wang et al, 2019a ), MARN (Pei et al, 2019 ), JSRL-VCT (Hou et al, 2019 ), GRU-EVE (Aafaq et al, 2019 ), STG-KD (Pan et al, 2020 ), SAAT (Zheng et al, 2020 ), and ORG-TRL (Zhang et al, 2020 ). Our method outperforms all the other methods on all the metrics by a large margin.…”

Section: Methodsmentioning

confidence: 99%

“…Table 2 displays the evaluation results of several video captioning models on the MSR-VTT. In this table, we compare our model with existing models, including MTVC (Pasunuru and Bansal, 2017a ), CIDEnt-RL (Pasunuru and Bansal, 2017b ), SibNet (Liu et al, 2018 ), HACA (Wang et al, 2018 ), TAMoE (Wang et al, 2019b ), POS (Wang et al, 2019a ), MARN (Pei et al, 2019 ), JSRL-VCT (Hou et al, 2019 ), GRU-EVE (Aafaq et al, 2019 ), STG-KD (Pan et al, 2020 ), SAAT (Zheng et al, 2020 ), ORG-TRL (Zhang et al, 2020 ). According to the overall score defined in (16), ORG-TRL is the best among existing models.…”

Section: Methodsmentioning

confidence: 99%

“…Inspired by Huszar ( 2015 ), who mathematically proves that both the Teacher Forcing algorithm and Curriculum Learning have a tendency to learn a biased model, Goyal et al ( 2016 ) solve the problem by adopting an adversarial domain method to align the dynamics of the RNN during training and inference. Zhang et al ( 2020 ) propose an object relational graph (ORG) to encode interaction features and design a teacher-recommended learning (TRL) method to utilize linguistic knowledge.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

A Semantics-Assisted Video Captioning Model Trained With Scheduled Sampling

et al. 2020

View full text Add to dashboard Cite

Given the features of a video, recurrent neural networks can be used to automatically generate a caption for the video. Existing methods for video captioning have at least three limitations. First, semantic information has been widely applied to boost the performance of video captioning models, but existing networks often fail to provide meaningful semantic features. Second, the Teacher Forcing algorithm is often utilized to optimize video captioning models, but during training and inference, different strategies are applied to guide word generation, leading to poor performance. Third, current video captioning models are prone to generate relatively short captions that express video contents inappropriately. Toward resolving these three problems, we suggest three corresponding improvements. First of all, we propose a metric to compare the quality of semantic features, and utilize appropriate features as input for a semantic detection network (SDN) with adequate complexity in order to generate meaningful semantic features for videos. Then, we apply a scheduled sampling strategy that gradually transfers the training phase from a teacher-guided manner toward a more self-teaching manner. Finally, the ordinary logarithm probability loss function is leveraged by sentence length so that the inclination of generating short sentences is alleviated. Our model achieves better results than previous models on the YouTube2Text dataset and is competitive with the previous best model on the MSR-VTT dataset.

show abstract

Section: Methodsmentioning

confidence: 90%

Section: Methodsmentioning

confidence: 99%

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

A Semantics-Assisted Video Captioning Model Trained With Scheduled Sampling

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Zhang et al [155] have proposed an Object Relational Graph with Teacher-Recommended Learning (ORG-TRL) system, which consists of an Object-Relational Graph (ORG) based encoder, which can capture more comprehensive interaction features to improve visual representation. In the data preprocessing phase, Inception-Resnet-v2 and C3D are used for visual and spatio-temporal feature extraction purposes.…”

Section: Feature Extractionmentioning

confidence: 99%

Exploring Video Captioning Techniques: A Comprehensive Survey on Deep Learning Methods

et al. 2021

View full text Add to dashboard Cite

Video captioning is an automated collection of natural language phrases that explains the contents in video frames. Because of the incomparable performance of deep learning in the field of computer vision and natural language processing in recent years, research in this field has been exponentially increased throughout past decades. Numerous approaches, datasets, and measurement metrics have been introduced in the literature, calling for a systematic survey to guide research efforts in this exciting new direction. Through the statistical analysis, this survey paper focuses mostly on state-of-the-art approaches, emphasizing deep learning models, assessing benchmark datasets in several parameters, and classifying the pros and cons of the various evaluation metrics based on the previous works in the deep learning field. This survey shows the most used variants of neural networks for visual and spatio-temporal feature extraction as well as language generation model. The results show that ResNet and VGG as visual feature extractor and 3D convolutional neural network as spatio-temporal feature extractor are mostly used. Besides that, Long Short Term Memory (LSTM) has been mainly used as the language model. However, nowadays, the Gated Recurrent Unit (GRU) and Transformer are slowly replacing LSTM. Regarding dataset usage, so far, MSVD and MSR-VTT are very much dominant due to be part of outstanding results among various captioning models. From 2015 to 2020, with all major datasets, some models such as, Inception-Resnet-v2 + C3D + LSTM, ResNet-101 + I3D + Transformer, ResNet-152 + ResNext-101 (R3D) + (LSTM, GAN) have achieved by far best results in video captioning. Despite rapid advancement, our survey reveals that video captioning research-work still has a lot to develop in accessing the full potential of deep learning for classifying and captioning a large number of activities, as well as creating large datasets covering diversified training video samples.

show abstract

Deep learning and knowledge graph for image/video captioning: A review of datasets, evaluation metrics, and methods

Wajid,

Terashima‐Marin,

Najafirad

et al. 2023

Engineering Reports

View full text Add to dashboard Cite

Generating an image/video caption has always been a fundamental problem of Artificial Intelligence, which is usually performed using the potential of Deep Learning Methods, Computer Vision, Knowledge Graphs, and Natural Language Processing (NLP). The significant task of image/video captioning is to describe visual content in terms of natural language. Due to a semantic gap, this presents a massive problem in understanding and explaining images or videos syntactically and semantically. The current systems need somewhere to fill the gap between low‐level and high‐level features while mapping. Therefore, to tackle this problem, there is a need to describe the latest research and methods to overcome difficulties and to propose effective solutions. This work thoroughly analyses and investigates the most related methods (deep learning and knowledge graph‐based approaches), benchmark datasets, and evaluation metrics with their benefits and limitations. Here we have also reviewed the state‐of‐the‐art methods related to image/video captioning and their applications in the current scenario. Finally, we provide thorough information on existing research with comparisons of results on benchmark datasets. We have also mentioned the existing challenges and future direction of research.

show abstract

Object Relational Graph With Teacher-Recommended Learning for Video Captioning

Cited by 226 publications

References 37 publications

A Semantics-Assisted Video Captioning Model Trained With Scheduled Sampling

A Semantics-Assisted Video Captioning Model Trained With Scheduled Sampling

Exploring Video Captioning Techniques: A Comprehensive Survey on Deep Learning Methods

Deep learning and knowledge graph for image/video captioning: A review of datasets, evaluation metrics, and methods

Contact Info

Product

Resources

About