Video Captioning via Sentence Augmentation and Spatio-Temporal Attention

Chen, Tseng-Hung; Zeng, Kuo-Hao; Hsu, Wan Ting; Sun, Min

doi:10.1007/978-3-319-54407-6_18

Cited by 5 publications

(2 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, at each word generation step, the decoder takes as input the video features weighted according to their relevance to the next word, based on the previously emitted words [31], [32], [38], [39]. With the same principle, in [40] the attention mechanism is applied to the mean-pooled features from a predefined number of objects tracklets in the video. In [41], the textual information is used to select Regionsof-Interest (ROIs) in the video frames, whose descriptors are combined with those of the global frame in a Dual Memory Recurrent Model.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

The Role of the Input in Natural Language Video Description

Cascianelli

Devo

Ciarfuglia

et al. 2020

IEEE Trans. Multimedia

View full text Add to dashboard Cite

Natural Language Video Description (NLVD) has recently received strong interest in the Computer Vision, Natural Language Processing (NLP), Multimedia, and Autonomous Robotics communities. The State-of-the-Art (SotA) approaches obtained remarkable results when tested on the benchmark datasets. However, those approaches poorly generalize to new datasets. In addition, none of the existing works focus on the processing of the input to the NLVD systems, which is both visual and textual. In this work, it is presented an extensive study dealing with the role of the visual input, evaluated with respect to the overall NLP performance. This is achieved performing data augmentation of the visual component, applying common transformations to model camera distortions, noise, lighting, and camera positioning, that are typical in real-world operative scenarios. A t-SNE based analysis is proposed to evaluate the effects of the considered transformations on the overall visual data distribution. For this study, it is considered the English subset of Microsoft Research Video Description (MSVD) dataset, which is used commonly for NLVD. It was observed that this dataset contains a relevant amount of syntactic and semantic errors. These errors have been amended manually, and the new version of the dataset (called MSVD-v2) is used in the experimentation. The MSVD-v2 dataset is released to help to gain insight into the NLVD problem.

show abstract

Section: Related Workmentioning

confidence: 99%

“…Few works on NLVD operate at input level. In [40] data augmentation is proposed at the sentence level. The authors proposed to enrich the sentence part of the MSR-VTT with sentences from the MSVD.…”

Section: Related Workmentioning

confidence: 99%