Distinctive-Attribute Extraction for Image Captioning

Abstract

Image captioning, an open research issue, has been evolved with the progress of deep neural networks. Convolutional neural networks (CNNs) and recurrent neural networks (RNNs) are employed to compute image features and generate natural language descriptions in the research. In previous works, a caption involving semantic description can be generated by applying additional information into the RNNs. In this approach, we propose a distinctive-attribute extraction (DaE) which explicitly encourages significant meanings to generate an accurate caption describing the overall meaning of the image with their unique situation. Specifically, the captions of training images are analyzed by term frequency-inverse document frequency (TF-IDF), and the analyzed semantic information is trained to extract distinctive-attributes for inferring captions. The proposed scheme is evaluated on a challenge data, and it improves an objective performance while describing images in more detail.

Image Captioning with Internal and External Knowledge

“…where f w is a function that calculates the weight value allocated to w i , x i is the embedding vector of w i , and s t represents the word context vector at time t. Note that the δ t k stays the same during the generation of each word until the last time step. Here, inspired by the previous works [16] [9], we use TF-IDF method as the function f w , as this method can measure the importance degree of each word in a sentence or document. The word context vector s t is then fused with the previous hidden state h t −1 of the LSTM decoder to combine more compact semantic information to guide the visual attention, calculated as follows:…”

Boost image captioning with knowledge reasoning

“…where f w is a function that calculates the weight value allocated to w i , x i is the embedding vector of w i , and s t represents the word context vector at time t. Note that the tk stays the same during the generation of each word until the last time step. Here, inspired by the previous works (Kim et al 2018;Park et al 2017), we use TF-IDF method as the function f w , as this method can measure the importance degree of each word in a sentence or document. The word context vector s t is then fused with the previous hidden state h t−1 of the LSTM decoder to combine more compact semantic information to guide the visual attention, calculated as follows:…”