Video Captioning With Adaptive Attention and Mixed Loss Optimization

Xiao, Huanhou; Shi, Jing

doi:10.1109/access.2019.2942000

Cited by 9 publications

(4 citation statements)

References 36 publications

(61 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, methods [4], [16] extract visual features from a video using CNNs, encode the video features, and then use LSTMs to decode them into natural language. To further improve the captioning quality, subsequent work extends the encoder-decoder structure by consolidating temporal attention mechanism [5], hierarchical RNNs [6], [8], LSTMs with visual-semantic embedding [7], semantic decoder [9], [10], [11], spatial-hard attention [31], reconstruction network [12], and reinforced adaptive attention [32]. Despite these efforts, they are limited by generating a single sentence from a trimmed video.…”

Section: Related Work a Video Captioning Tasks 1) Video Captioningmentioning

confidence: 99%

Step by Step: A Gradual Approach for Dense Video Captioning

2023

View full text Add to dashboard Cite

Dense video captioning aims to localize and describe events for storytelling in untrimmed videos. It is a conceptually very challenging task that requires concise, relevant, and coherent captioning based on high-quality event localization. Unlike simple temporal action localization tasks without overlapping events, dense video captioning requires detecting multiple/overlapping regions in order to branch out the video story. Most existing methods generate numerous candidate event proposals and then eliminate duplicate ones using a event proposal selection algorithm (e.g., non-maximum suppression) or generate event proposals directly through box prediction and binary classification mechanisms, similar to object detection tasks. Despite these efforts, the aforementioned approaches tend to fail to localize overlapping events into different stories, hindering high-quality captioning. In this paper, we propose SBS, a dense video captioning framework with a gradual approach that addresses the challenge of localizing overlapping events and eventually constructs high-quality captioning. SBS accurately estimates the number of explicit events for each video snippet and then detects the boundaries context/activities, which are the details for generating the event proposals. Based on both the number of events and boundaries, SBS generates the event proposals. SBS encodes the context of the event sequence and finally generates sentences describing the event proposals. Our framework is fairly effective in localizing multiple/overlapping events, thus experimental results show the state-of-the-art performance compared to the existing methods.INDEX TERMS Dense video captioning, event captioning, event localization, event proposal generation, video captioning.

show abstract

Section: Related Work a Video Captioning Tasks 1) Video Captioningmentioning

confidence: 99%

Step by Step: A Gradual Approach for Dense Video Captioning

2023

View full text Add to dashboard Cite

show abstract

“…For example, basic frameworks [4], [13] capture visual features from a video using CNNs, then encode the video features and decode them into natural language using LSTMs. To improve the captioning quality, subsequent work has extended the basic encoder-decoder structure by incorporating a temporal attention mechanism [5], hierarchical RNNs [6], [8], LSTMs with visual-semantic embedding [7], a semantic decoder [9]- [11], reconstruction network [12], spatial-hard attention [14], and reinforced adaptive attention [15]. Despite a lot of effort, they are limited to generating a single sentence from a video.…”

Section: A Video Captioningmentioning

confidence: 99%

Parallel Pathway Dense Video Captioning With Deformable Transformer

2022

View full text Add to dashboard Cite

Dense video captioning is a very challenging task because it requires a high-level understanding of the video story, as well as pinpointing details such as objects and motions for a consistent and fluent description of the video. Many existing solutions divide this problem into two sub-tasks, event detection and captioning, and solve them sequentially ("localize-then-describe" or reverse). Consequently, the final outcome is highly dependent on the performance of the preceding modules. Detecting events or generating sentences directly from vast amounts of video information is prone to loss of elaborateness (accuracy and fluency), and hence propagates as a cascade effect to subsequent modules. Parallel decoding approaches have been proposed to overcome these limitations, which leads to another challenge: the bottleneck at the branch point. In this paper, we decompose this sequential approach by proposing a parallel pathway dense video captioning framework (PPVC) that localizes and describes events simultaneously without any bottlenecks. We introduce a representation organization network at the branching point of the parallel pathway to organize the encoded video feature by considering the entire storyline. Then, an event localizer focuses to localize events without any event proposal generation network, a sentence generator describes events while considering the fluency and coherency of sentences. A gating network controls the flow of organized representations to filter out unnecessary information. Our method has several advantages over existing work: (i) the final output does not depend on the output of the preceding modules, (ii) it improves existing parallel decoding methods by relieving the bottleneck of information. We evaluate the performance of PPVC on large-scale benchmark datasets, the ActivityNet Captions, and YouCook2. PPVC not only outperforms existing algorithms on the majority of metrics but also improves on both datasets by 5.4% and 4.9% compared to the state-of-the-art parallel decoding method.INDEX TERMS Machine learning, deep learning, video and language, video captioning I. INTRODUCTION

show abstract

“…At that point, the visual and semantic stream together produced an important textual description from a visual scene. Another approach that included adaptive attention and mixed loss optimization for performing video captioning was presented by Xiao et al [21]. The technique included a reinforced adaptive attention mechanism.…”

Section: End-to-end Modelmentioning

confidence: 99%

Att-BiL-SL: Attention-Based Bi-LSTM and Sequential LSTM for Describing Video in the Textual Formation

et al. 2021

View full text Add to dashboard Cite

With the advancement of the technological field, day by day, people from around the world are having easier access to internet abled devices, and as a result, video data is growing rapidly. The increase of portable devices such as various action cameras, mobile cameras, motion cameras, etc., can also be considered for the faster growth of video data. Data from these multiple sources need more maintenance to process for various usages according to the needs. By considering these enormous amounts of video data, it cannot be navigated fully by the end-users. Throughout recent times, many research works have been done to generate descriptions from the images or visual scene recordings to address the mentioned issue. This description generation, also known as video captioning, is more complex than single image captioning. Various advanced neural networks have been used in various studies to perform video captioning. In this paper, we propose an attention-based Bi-LSTM and sequential LSTM (Att-BiL-SL) encoder-decoder model for describing the video in textual format. The model consists of two-layer attention-based bi-LSTM and one-layer sequential LSTM for video captioning. The model also extracts the universal and native temporal features from the video frames for smooth sentence generation from optical frames. This paper includes the word embedding with a soft attention mechanism and a beam search optimization algorithm to generate qualitative results. It is found that the architecture proposed in this paper performs better than various existing state of the art models.

show abstract

Video Captioning With Adaptive Attention and Mixed Loss Optimization

Cited by 9 publications

References 36 publications

Step by Step: A Gradual Approach for Dense Video Captioning

Step by Step: A Gradual Approach for Dense Video Captioning

Parallel Pathway Dense Video Captioning With Deformable Transformer

Att-BiL-SL: Attention-Based Bi-LSTM and Sequential LSTM for Describing Video in the Textual Formation

Contact Info

Product

Resources

About