Image captioning via hierarchical attention mechanism and policy gradient optimization

Yan, Shiyang; Xie, Yuan; Wu, Frederick Y.; Smith, Jeremy S.; Lu, Wenjin; Zhang, Bailing

doi:10.1016/j.sigpro.2019.107329

Cited by 41 publications

(21 citation statements)

References 44 publications

(69 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This study found many articles in which the results of their models had been compared with state of the art models, such as refs. [2][3][4][5]. As these models were built some years ago, they have been more cited so are easier to find during a search on the digital libraries.…”

Section: Comparison Of Resultsmentioning

confidence: 99%

“…[1]. Four successful articles [2][3][4][5], which now are the most cited articles researchers rely on, were published in 2015. There was not much interest in this area in 2014 and 2015, but it is clear from this review how exponentially the popularity is growing-57 articles found were published in 2017-2018 and already 17 were published during the first three months of 2019.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

A Systematic Literature Review on Image Captioning

Staniūtė

Šešok

2019

Applied Sciences

View full text Add to dashboard Cite

Natural language problems have already been investigated for around five years. Recent progress in artificial intelligence (AI) has greatly improved the performance of models. However, the results are still not sufficiently satisfying. Machines cannot imitate human brains and the way they communicate, so it remains an ongoing task. Due to the increasing amount of information on this topic, it is very difficult to keep on track with the newest researches and results achieved in the image captioning field. In this study a comprehensive Systematic Literature Review (SLR) provides a brief overview of improvements in image captioning over the last four years. The main focus of the paper is to explain the most common techniques and the biggest challenges in image captioning and to summarize the results from the newest papers. Inconsistent comparison of results achieved in image captioning was noticed during this study and hence the awareness of incomplete data collection is raised in this paper. Therefore, it is very important to compare results of a newly created model produced with the newest information and not only with the state of the art methods. This SLR is a source of such information for researchers in order for them to be precisely correct on result comparison before publishing new achievements in the image caption generation field.

show abstract

Section: Comparison Of Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

A Systematic Literature Review on Image Captioning

Staniūtė

Šešok

2019

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…The neural audio caption method based on data transformation can generate understandable and appropriate evaluations. To make the generated comment closer to the expert comment, a Generative Adversarial Network (GAN) is combined [33,34].…”

Section: Generative Adversarial Network-based Neural Audio Caption Modelmentioning

confidence: 99%

Generative Adversarial Network-Based Neural Audio Caption Model for Oral Evaluation

et al. 2020

View full text Add to dashboard Cite

Oral evaluation is one of the most critical processes in children’s language learning. Traditionally, the Scoring Rubric is widely used in oral evaluation for providing a ranking score by assessing word accuracy, phoneme accuracy, fluency, and accent position of a tester. In recent years, by the emerging demands of the market, oral evaluation requires not only providing a single score from pronunciation but also in-depth, meaning comments based on content, context, logic, and understanding. However, the Scoring Rubric requires massive human work (oral evaluation experts) to provide such deep meaning comments. It is considered uneconomical and inefficient in the current market. Therefore, this paper proposes an automated expert comment generation approach for oral evaluation. The approach first extracts the oral features from the children’s audio as well as the text features from the corresponding expert comments. Then, a Gated Recurrent Unit (GRU) is applied to encode the oral features into the model. Afterwards, a Long Short-Term Memory (LSTM) model is applied to train the mappings between oral features and text features and generate expert comments for the new coming oral audio. Finally, a Generative Adversarial Network (GAN) is combined to improve the quality of the generated comments. It generates pseudo-comments to train the discriminator to recognize the human-like comments. The proposed approach is evaluated in a real-world audio dataset (children oral audio) collected by our collaborative company. The proposed approach is also integrated into a commercial application to generate expert comments for children’s oral evaluation. The experimental results and the lessons learned from real-world applications show that the proposed approach is effective for providing meaningful comments for oral evaluation.

show abstract

“…Generating human-like captions for images automatically, namely, image captioning, has risen as an interdisciplinary research issue at the crossing point of computer vision and natural language processing [1,2,3,4,5,6,7]. It has numerous imperative industrial applications, such as assistant facilities for visually impaired individuals, visual knowledge in chatting robots, and photo sharing on social media.…”

Section: Introductionmentioning

confidence: 99%

Diverse and Styled Image Captioning Using SVD-Based Mixture of Recurrent Experts

Heidari,

Ghatee,

Nickabadi

et al. 2020

Preprint

View full text Add to dashboard Cite

With great advances in vision and natural language processing, the generation of image captions becomes a need. In a recent paper, Mathews, Xie and He [1], extended a new model to generate styled captions by separating semantics and style. In continuation of this work, here a new captioning model is developed including an image encoder to extract the features, a mixture of recurrent networks to embed the set of extracted features to a set of words, and a sentence generator that combines the obtained words as a stylized sentence. The resulted system that entitled as Mixture of Recurrent Experts (MoRE), uses a new training algorithm that derives singular value decomposition (SVD) from weighting matrices of Recurrent Neural Networks (RNNs) to increase the diversity of captions. Each decomposition step depends on a distinctive factor based on the number of RNNs in MoRE. Since the used sentence generator gives a stylized language corpus without paired images, our captioning model can do the same. Besides, the styled and diverse captions are extracted without training on a densely labeled or styled dataset. To validate this captioning model, we use Microsoft COCO which is a standard factual image caption dataset. We show that the proposed captioning model can generate a diverse and stylized image captions without the necessity of extra-labeling. The results also show better descriptions in terms of content accuracy.

show abstract

Image captioning via hierarchical attention mechanism and policy gradient optimization

Cited by 41 publications

References 44 publications

A Systematic Literature Review on Image Captioning

A Systematic Literature Review on Image Captioning

Generative Adversarial Network-Based Neural Audio Caption Model for Oral Evaluation

Diverse and Styled Image Captioning Using SVD-Based Mixture of Recurrent Experts

Contact Info

Product

Resources

About