Connecting What to Say With Where to Look by Modeling Human Attention Traces

Abstract

We introduce a unified framework to jointly model images, text, and human attention traces. Our work is built on top of the recent Localized Narratives annotation framework [30], where each word of a given caption is paired with a mouse trace segment. We propose two novel tasks:(1) predict a trace given an image and caption (i.e., visual grounding), and (2) predict a caption and a trace given only an image. Learning the grounding of each word is challenging, due to noise in the human-provided traces and the presence of words that cannot be meaningfully visually grounded. We present a novel model architecture that is jointly trained on dual tasks (controlled trace generation and controlled caption generation). To evaluate the quality of the generated traces, we propose a local bipartite matching (LBM) distance metric which allows the comparison of two traces of different lengths. Extensive experiments show our model is robust to the imperfect training data and outperforms the baselines by a clear margin. Moreover, we demonstrate that our model pre-trained on the proposed tasks can be also beneficial to the downstream task of COCO's guided image captioning. Our code 1 and project page 2 are publicly available.

PathNarratives: Data annotation for pathological human-AI collaborative diagnosis

“…Narrative annotation focuses on the description of the relationship between entities, and entity relationships are collected during the annotation phase. Attributes, relationships, and entities in the same image are often closely related (29)(30)(31)(32). Localized Narratives (30) connect vision and language by artificially using mouse scribing to join action connections between entities and make the captioning in content more hierarchical.…”

Human-Centric Image Retrieval with Gaze-Based Image Captioning

“…In order to generate the word-to-box alignment from the provided gaze trace points, our model divides the trace into several boxes, each box associated with a word, and generates a bounding box aligned on the axis of the gaze points. This trace transformation is performed inspired by the mouse trace analysis [17]. After that we extract three kinds of features, visual, captioning and gaze features.…”

“…After that we extract three kinds of features, visual, captioning and gaze features. For calculating visual features, we use pre-trained Faster R-CNN [17] provided by detectron2 [18] to compute the visual features of the detected regions. Next, for calculating captioning features, we sum up the positional embeddings and the word embeddings via LXMERT proposed in the previous method [19].…”

“…The CGT model, which includes three modules, image encoder, caption encoder-decoder and gaze trace encoder-decoder, is constructed based on the previous method [17]. Figure 2 shows the overview of our proposed CGT model.…”

“…λ * are the balance parameters between all loss functions. This settings are based on the previous method [17].…”