Using human experts' gaze data to evaluate image processing algorithms

Vaidyanathan, Preethi; Pelz, Jeff B.; Li, Rui; Mulpuru, Sai; Wang, Dong; Shi, Peng; Calvelli, Cara; Haake, Anne R.

doi:10.1109/ivmspw.2011.5970367

Cited by 7 publications

(7 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The expert dermatologist group was instructed to "examine and describe each image verbally as if teaching the trainee to make a diagnosis based on the image." Physician assistant students were recruited to serve as 'trainees' in order to motivate the expert dermatologists through the modified MasterApprentice model [Beyer and Holtzblatt 1997;Vaidyanathan et al 2011]. We also recorded experts' verbal narration.…”

Section: Methodsmentioning

confidence: 99%

“…For example, in addition to perceiving the presence of certain cues, they also perceive the absence of other critical cues, performing meaningful integration of that information. Characterizing experts' perceptual and conceptual expertise by capturing their viewing behavior and spoken description * pxv1621@g.rit.edu will improve our understanding of how experts perform such complex cognitive tasks using their domain knowledge, and this will benefit image informatics systems [Vaidyanathan et al 2011;Li et al 2012]. This work focuses on how the RQA method [Anderson et al 2013] can be extended to understand the effect of perceptual expertise on eye movement patterns and its potential use in investigating the interactions between experts' eye movements and spoken description.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Recurrence quantification analysis reveals eye-movement behavior differences between experts and novices

Vaidyanathan

Pelz

Alm

et al. 2014

Proceedings of the Symposium on Eye Tracking Research and Applications

Self Cite

View full text Add to dashboard Cite

Understanding and characterizing perceptual expertise is a major bottleneck in developing intelligent systems. In knowledge-rich domains such as dermatology, perceptual expertise influences the diagnostic inferences made based on the visual input. This study uses eye movement data from 12 dermatology experts and 12 undergraduate novices while they inspected 34 dermatological images. This work investigates the differences in global and local temporal fixation patterns between the two groups using recurrence quantification analysis (RQA). The RQA measures reveal significant differences in both global and local temporal patterns between the two groups. Results show that experts tended to refixate previously inspected areas less often than did novices, and their refixations were more widely separated in time. Experts were also less likely to follow extended scan paths repeatedly than were novices. These results suggest the potential value of RQA measures in characterizing perceptual expertise. We also discuss potential use of the RQA method in understanding the interactions between experts' visual and linguistic behavior.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Recurrence quantification analysis reveals eye-movement behavior differences between experts and novices

Vaidyanathan

Pelz

Alm

et al. 2014

Proceedings of the Symposium on Eye Tracking Research and Applications

Self Cite

View full text Add to dashboard Cite

show abstract

“…Extending Vaidyanathan et al [2015a], we also include the K-means method to encode fixation sequences. We use K-means and Lab color features as they have been particularly useful for dermatological images [Bosman et al 2010;Vaidyanathan et al 2011]. Each image is first converted into Lab color space, where the L channel represents illumination, the a channel indicates redness-greenness, and the b channel indicates blueness-yellowness in the image.…”

Section: Visual Unitsmentioning

confidence: 99%

Fusing eye movements and observer narratives for expert-driven image-region annotations

Vaidyanathan¹,

Prud’hommeaux

Pelz³

et al. 2016

Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research &Amp; Applications

Self Cite

View full text Add to dashboard Cite

Human image understanding is reflected by individuals' visual and linguistic behaviors, but the meaningful computational integration and interpretation of their multimodal representations remain a challenge. In this paper, we expand a framework for capturing image-region annotations in dermatology, a domain in which interpreting an image is influenced by experts' visual perception skills, conceptual domain knowledge, and task-oriented goals. Our work explores the hypothesis that eye movements can help us understand experts' perceptual processes and that spoken language descriptions can reveal conceptual elements of image inspection tasks. We cast the problem of meaningfully integrating visual and linguistic data as unsupervised bitext alignment. Using alignment, we create meaningful mappings between physicians' eye movements, which reveal key areas of images, and spoken descriptions of those images. The resulting alignments are then used to annotate image regions with medical concept labels. Our alignment accuracy exceeds baselines using both exact and delayed temporal correspondence. Additionally, comparison of alignment accuracy between a method that identifies clusters in the images based on eye movement vs. a method that identifies clusters using image features suggests that the two approaches perform well on different types of images and concept labels. This suggests that an image annotation framework should integrate information from more than one technique to handle heterogeneous images. We also investigate the performance of the proposed aligner for dermatological primary morphology concept labels, as well as for lesion size or type and distribution-based categories of images.

show abstract

“…The device of capture is usually a physical piece of equipment, which records a measurable human output. For example, eye movements and related measurements of an observer can be objectively captured with adequate fidelity by an eye-tracker under the right conditions (Vaidyanathan et al 2011;Wang et al 2013), and an oximeter can be used to measure a person's blood oxygen level and pulse (Bethamcherla et al 2015). Similarly, linguistic sensing by definition involves capturing and objectively measuring the linguistic signal response produced by a language user.…”

Section: Introductionmentioning

confidence: 99%

Language as Sensor in Human‐Centered Computing: Clinical Contexts as Use Cases

Alm

2016

Language and Linguist. Compass

View full text Add to dashboard Cite

Recognizing the increased emphasis on adaptive and personalized computing, as well as the popularity of multimodal natural language and speech processing in human-centered applications, this paper articulates and discusses the role of natural language as a rich and meaningful sensor of the human body and mind. As the primary means of human interaction, information sharing, and social, emotional, and interpersonal grounding, linguistic sensing provides a particularly useful and rich sensory window for understanding the dynamically and temporally unfolding state of the human body and mind for non-linguistic aims. For sensory monitoring, linguistic data are unobtrusive, inexpensive, pervasive, natural, especially insightful from the perspective of cognitive processing, and generally convenient to capture, in contrast to many other sensor modalities that require excessive wearable equipment or smart clothing. Using clinical computational linguistics as a springboard for discussing and exemplifying this unconventional yet increasingly prevalent view of language, including its role in practical and serviceable linguistic analysis, this overview surveys recent work in computational linguistics that broadly takes advantage of linguistic sensing's potential. The discussion also outlines existing challenges and considerations for continued research in this area, from the perspective of natural language as a sensor in human-centered computing and the development of innovative and applied language technologies.

show abstract

Using human experts' gaze data to evaluate image processing algorithms

Cited by 7 publications

References 10 publications

Recurrence quantification analysis reveals eye-movement behavior differences between experts and novices

Recurrence quantification analysis reveals eye-movement behavior differences between experts and novices

Fusing eye movements and observer narratives for expert-driven image-region annotations

Language as Sensor in Human‐Centered Computing: Clinical Contexts as Use Cases

Contact Info

Product

Resources

About