Densely Supervised Hierarchical Policy-Value Network for Image Paragraph Generation

Abstract

Image paragraph generation aims to describe an image with a paragraph in natural language. Compared to image captioning with a single sentence, paragraph generation provides more expressive and fine-grained description for storytelling. Existing approaches mainly optimize paragraph generator towards minimizing word-wise cross entropy loss, which neglects linguistic hierarchy of paragraph and results in ``sparse" supervision for generator learning. In this paper, we propose a novel Densely Supervised Hierarchical Policy-Value (DHPV) network for effective paragraph generation. We design new hierarchical supervisions consisting of hierarchical rewards and values at both sentence and word levels. The joint exploration of hierarchical rewards and values provides dense supervision cues for learning effective paragraph generator. We propose a new hierarchical policy-value architecture which exploits compositionality at token-to-token and sentence-to-sentence levels simultaneously and can preserve the semantic and syntactic constituent integrity. Extensive experiments on the Stanford image-paragraph benchmark have demonstrated the effectiveness of the proposed DHPV approach with performance improvements over multiple state-of-the-art methods.

Hierarchical Scene Graph Encoder-Decoder for Image Paragraph Captioning

“…We compare our HSGED(SLL) with several state-of-the-art models: Regions-Hierarchical [13], RTT-GAN [17], DCPG [5], HCAVP [46], DHPV [36], CAE-LSTM [34], TDPG [23] and CRL [22]. Among these methods, RTT-GAN, DCPG, Regions-Hierarchical, DHPV, Hierarchical CAVP and CAE-LSTM use HRNNs with different technique details.…”

“…Researchers also propose advanced techniques to refine the prototypical HRNN, e.g., generative models like GAN [17] or VAE [5] for stronger consistency; the trigram repetition penalty based sampling method for diversity [23]. Besides, dense sentencelevel rewards [36] and curiosity-driven reinforcement learning [22] are used for more robust training, all of which could also be applied in our proposed framework, HSGED. However, most of them are built without enough hierarchical constraints, so the qualities of the generated paragraphs are unsatisfactory.…”

“…In this way, a more informative paragraph can be generated, as in Figure 1 (d), the generated sentences cover more topics than those generated by the flat RNN in Figure 1 (c). However, the multi-level RNNs in HRNN are built without any hierarchical constraints [5,13,17,36,46], which results in two problems. 1) The topics are not coherent and distinctive since they are formed by randomly sampling some object sub-sets without any global constraint.…”

Geo-Aware Image Caption Generation

“…Our GeoRic dataset consists of 29,038 images from the Geograph project website, with captions and location coordinates. We selected captions that are exactly one sentence long (multi-sentence caption generation, although a promising research direction (Mao et al, 2018;Wu et al, 2019), is not addressed in this work) and include at least one spatial expression, such as "near", "north of", "across", etc. (in order to ensure that the captions contain enough geographic referencing).…”

Incorporating additional knowledge into image captioners

“…Besides, dense sentence-level rewards [49] and curiosity-driven reinforcement learning [50] are used for more robust training, all of which could also be applied in our proposed framework, HSGED. However, most of them are built without enough 2.3.…”