Learning Long- and Short-Term User Literal-Preference with Multimodal Hierarchical Transformer Network for Personalized Image Caption

Abstract

Personalized image caption, a natural extension of the standard image caption task, requires to generate brief image descriptions tailored for users' writing style and traits, and is more practical to meet users' real demands. Only a few recent studies shed light on this crucial task and learn static user representations to capture their long-term literal-preference. However, it is insufficient to achieve satisfactory performance due to the intrinsic existence of not only long-term user literal-preference, but also short-term literal-preference which is associated with users' recent states. To bridge this gap, we develop a novel multimodal hierarchical transformer network (MHTN) for personalized image caption in this paper. It learns short-term user literal-preference based on users' recent captions through a short-term user encoder at the low level. And at the high level, the multimodal encoder integrates target image representations with short-term literal-preference, as well as long-term literal-preference learned from user IDs. These two encoders enjoy the advantages of the powerful transformer networks. Extensive experiments on two real datasets show the effectiveness of considering two types of user literal-preference simultaneously and better performance over the state-of-the-art models.

Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation

“…This task requires the model to generate captions to describe the content of images [59]. State-of-the-art methods follow multi-modal attention designs, treating the task as a multi-modal translation problem [66,73,75]. Our focus in this work is not to design a new captioning model, but to explore image captioning as a sub-task for open vocabulary learning to enhance the novel class discovery ability.…”

From Show to Tell: A Survey on Deep Learning-based Image Captioning

“…To this end, early approaches exploit a memory block as a repository for this contextual information [207], [208]. On another line, Zhang et al [209] proposed a multi-modal Transformer network that personalizes captions conditioned on the user's recent captions and a learned user representation. Other works have instead focused on the style of captions as an additional controllable input and proposed to solve this task by exploiting unpaired stylized textual corpus [210], [211], [212], [213] and adversarial learning [212].…”

Infrared Small-Dim Target Detection with Transformer under Complex Backgrounds

“…Inspired by the success of the self-attention mechanism adopted in transformer architectures in the NLP field [10,11], some works employed them in the computer vision field [35][36][37].…”