Speak to your Software Visualization—Exploring Component-Based Software Architectures in Augmented Reality with a Conversational Interface

Abstract

Exploring of software architectures with software visualization in Augmented Reality (AR) is possible with different interaction methods, such gesture, gaze, and speech. For interaction with speech (i.e., natural language), we present an architecture and an implementation of conversational interfaces for the Microsoft HoloLens device. We aim to remedy some peculiarities of AR devices, but also enhancing the exploration task at hand. To implement the conversational interface different natural language processing (NLP) components such as natural language generation and intent recognition are typically required. Our proposed architecture integrates conversational components with the AR-based software visualization. We describe its implementation based on different user utterances, where the system provides information about the to-be-explored component-based software architecture in the form of adjusted visualizations and speech-based results. We apply out tool to explore OSGi-based software architectures.

A Detailed Study on Aggregation Methods used in Natural Language Interface to Databases (NLIDB)

“…Seipel, et al [30] implemented a conversational interface architecture for the Microsoft HoloLens device. By addressing some of the drawbacks of AR technology, this study hopes to enhance the exploration process.…”

Towards Natural Language Interfaces for Data Visualization: A Survey

“…Bieliauskas et al [20] proposed an interaction approach with software visualizations based on a conversational interface, like Orko [234], it can automatically display best fitting views according to meta information from natural language sentences. Seipel et al [203] explored natural language interaction for software visualization in Augmented Reality (AR) with Microsoft HoloLens device, which can provide various view transformation interactions for users through gesture, gaze, and speech. In general, view transformations combined with V-NLI are currently mainly focused on viewpoint navigation.…”

“…In the visualization community, several works have augmented static visualizations with virtual content [36], [124]. Seipel et al [203] explored NLI for software visualization with AR devices. Reinders et al [191] studied blind and low vision (BLV) people's preference when exploring interactive 3D printed models (I3Ms).…”

Identifying Usability Issues of Software Analytics Applications in Immersive Augmented Reality

“…Reipschläger et al [10] present DebugAR, a tool to debug software systems in AR. It has also been studied, how to control the exploration of software architectures in AR through speech [11]- [13]. Mehra et al [14] present XRaSE, a tool to create immersive representations of software systems to support comprehension.…”