Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification

Kumar, Abhinav; Aurisano, Jillian; Eugenio, Barbara Di; Johnson, Andrew; Alsaiari, Abeer; Flowers, Nigel; González, Alberto; Leigh, Jason

doi:10.21437/semdial.2017-5

Cited by 11 publications

(8 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some recent work has started investigating the potential of building dialogue systems that can help users efficiently explore data through visualizations (Kumar et al, 2017).…”

Section: Related Workmentioning

confidence: 99%

Conversational Image Editing: Incremental Intent Identification in a New Dialogue Task

Manuvinakurike

Bui

Chang

et al. 2018

Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue

View full text Add to dashboard Cite

We present "conversational image editing", a novel real-world application domain combining dialogue, visual information, and the use of computer vision. We discuss the importance of dialogue incrementality in this task, and build various models for incremental intent identification based on deep learning and traditional classification algorithms. We show how our model based on convolutional neural networks outperforms models based on random forests, long short term memory networks, and conditional random fields. By training embeddings based on image-related dialogue corpora, we outperform pre-trained out-of-the-box embeddings, for intention identification tasks. Our experiments also provide evidence that incremental intent processing may be more efficient for the user and could save time in accomplishing tasks.

show abstract

“…Some recent work has started investigating the potential of building dialogue systems that can help users efficiently explore data through visualizations (Kumar et al, 2017).…”

Section: Related Workmentioning

confidence: 99%

Conversational Image Editing: Incremental Intent Identification in a New Dialogue Task

Manuvinakurike

Bui

Chang

et al. 2018

Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue

View full text Add to dashboard Cite

show abstract

“…Our work complements this analysis by observing similar tasks that spanned many views and utilized large display areas. Our work expands upon Aurisano et al [AKG*b, AKG*a], which presents preliminary analysis of this data, and Kumar et al [KADE*16, KDEA*], which examines utterances and gestures from a natural language processing perspective.…”

Section: Related Workmentioning

confidence: 88%

Many At Once: Capturing Intentions to Create And Use Many Views At Once In Large Display Environments

Aurisano

Kumar

Alsaiari

et al. 2020

Computer Graphics Forum

Self Cite

View full text Add to dashboard Cite

This paper describes results from an observational, exploratory study of visual data exploration in a large, multi‐view, flexible canvas environment. Participants were provided with a set of data exploration sub‐tasks associated with a local crime dataset and were instructed to pose questions to a remote mediator who would respond by generating and organizing visualizations on the large display. We observed that participants frequently posed requests to cast a net around one or several subsets of the data or a set of data attributes. They accomplished this directly and by utilizing existing views in unique ways, including by requesting to copy and pivot a group of views collectively and posing a set of parallel requests on target views expressed in one command. These observed actions depart from multi‐view flexible canvas environments that typically provide interfaces in support of generating one view at a time or actions that operate on one view at a time. We describe how participants used these ‘cast‐a‐net’ requests for tasks that spanned more than one view and describe design considerations for multi‐view environments that would support the observed multi‐view generation actions.

show abstract

“…To make multimodal interaction more smooth, the system should have a declarative representation for all these potential referents and find the best match. Articulate2 [120] addresses this issue by leveraging Kinect to detect deictic gestures on the virtual touch screen in front of a large display. If referring expressions are detected and a gesture has been detected by Kinect, information about any objects pointed to by the user will be stored and then the system can find the best match between properties of each relevant entity.…”

Section: Co-reference Resolutionmentioning

confidence: 99%

Towards Natural Language Interfaces for Data Visualization: A Survey

Shen,

Luo

et al. 2021

Preprint

View full text Add to dashboard Cite

Utilizing Visualization-oriented Natural Language Interfaces (V-NLI) as a complementary input modality to direct manipulation for visual analytics can provide an engaging user experience. It enables users to focus on their tasks rather than worrying about operating the interface to visualization tools. In the past two decades, leveraging advanced natural language processing technologies, numerous V-NLI systems have been developed both within academic research and commercial software, especially in recent years. In this article, we conduct a comprehensive review of the existing V-NLIs. In order to classify each paper, we develop categorical dimensions based on a classic information visualization pipeline with the extension of a V-NLI layer. The following seven stages are used: query understanding, data transformation, visual mapping, view transformation, human interaction, context management, and presentation. Finally, we also shed light on several promising directions for future work in the community.

show abstract

Multimodal Coreference Resolution for Exploratory Data Visualization Dialogue: Context-Based Annotation and Gesture Identification

Cited by 11 publications

References 17 publications

Conversational Image Editing: Incremental Intent Identification in a New Dialogue Task

Conversational Image Editing: Incremental Intent Identification in a New Dialogue Task

Many At Once: Capturing Intentions to Create And Use Many Views At Once In Large Display Environments

Towards Natural Language Interfaces for Data Visualization: A Survey

Contact Info

Product

Resources

About