Fusing eye movements and observer narratives for expert-driven image-region annotations

Abstract

Human image understanding is reflected by individuals' visual and linguistic behaviors, but the meaningful computational integration and interpretation of their multimodal representations remain a challenge. In this paper, we expand a framework for capturing image-region annotations in dermatology, a domain in which interpreting an image is influenced by experts' visual perception skills, conceptual domain knowledge, and task-oriented goals. Our work explores the hypothesis that eye movements can help us understand experts' perceptual processes and that spoken language descriptions can reveal conceptual elements of image inspection tasks. We cast the problem of meaningfully integrating visual and linguistic data as unsupervised bitext alignment. Using alignment, we create meaningful mappings between physicians' eye movements, which reveal key areas of images, and spoken descriptions of those images. The resulting alignments are then used to annotate image regions with medical concept labels. Our alignment accuracy exceeds baselines using both exact and delayed temporal correspondence. Additionally, comparison of alignment accuracy between a method that identifies clusters in the images based on eye movement vs. a method that identifies clusters using image features suggests that the two approaches perform well on different types of images and concept labels. This suggests that an image annotation framework should integrate information from more than one technique to handle heterogeneous images. We also investigate the performance of the proposed aligner for dermatological primary morphology concept labels, as well as for lesion size or type and distribution-based categories of images.

Computational framework for fusing eye movements and spoken narratives for image annotation

“…In this section, we describe our multimodal Spoken Narratives And Gaze ( snag ) dataset ( Vaidyanathan et al., 2018 ) that is used to evaluate the proposed framework. This dataset contains eye movements and spoken narratives co-captured from participants while viewing general domain images ( Figure 3) and has been released 2 to the research community.…”

“…Reference alignments (ground truth) were prepared using a GUI called RegionLabeler 5 ( Vaidyanathan et al., 2018 ) to allow evaluation of the resulting multimodal alignments. This represented the manual alignments obtained by associating each fixation cluster in the case of MFSC and image segment in the case of image segmentation with its corresponding word tokens (linguistic units).…”

“…This work integrates gaze and linguistic information indicating ‘what people look at’ and ‘what people say,’ to identify the objects and their corresponding names or labels in images. The data we collected ( Vaidyanathan et al., 2018 ) which has been released for research purposes and the code we developed for the framework (released in this work), allowed us to explore the following research questions: When a person views and describes an image, what relationship, if any, exists between the moment of fixation on an object and the moment the person utters the word or phrase to name that object? Can co-captured gaze and speech data be integrated automatically in order to identify and quantify this relationship?…”

SNAG: Spoken Narratives and Gaze Dataset

“…We examine the usefulness of our general-domain dataset on image-region annotation, adapting the framework given by Vaidyanathan et al (2016).…”

“…Ho et al (2015) provide a dataset that consists only of gaze and speech time stamps during dyadic interactions. The closest dataset to ours is the multimodal but non-public data described by Vaidyanathan et al (2016).…”