Multisensory Immersive Analytics

Abstract

While visual cues are traditionally used for visual analytics, multimodal interaction technologies offer many new possibilities. This chapter explores the opportunities and challenges for developers and users to utilize and represent data through non-visual sensory channels to help them understand and interact with data. Users are able to experience data in new ways: variables from complex datasets can be conveyed through different senses; presentations are more accessible to people with vision impairment and can be personalized to specific user needs; interactions can involve multiple senses to provide natural and transparent methods. All these techniques enable users to obtain a better understanding of the underlying information. While the emphasis of this chapter is towards non-visual immersive analytics, we include a discussion on how visual presentations are integrated with different modalities, and the opportunities of mixing several sensory signals, including the visual domain.

Grand Challenges in Immersive Analytics

“…A challenge complimentary to multimodal interaction is multisensory feedback. While multi-sensory data representation in Immersive Analytics has been previously discussed [91], there is a need to also consider multi-sensory output as feedback for input. Of the 5 senses (sight, sound, touch, smell, and taste), electronic devices usually are able to provide for sight and sound, though recent advances may facilitate other senses [103,110,133].…”

One View Is Not Enough: Review of and Encouragement for Multiple and Alternative Representations in 3D and Immersive Visualisation

“…In particular, especially due to the price drop of head-mounted displays (HMD), many researchers have explored how to visualise data beyond the use of a traditional desktop interface [8,9]. Subsequently, many areas are growing, including immersive analytics (IA) [3,10,11], multisensory visualisation [12], haptic data visualisation (HDV) [13], augmented visualisation [14], and olfactory visualisation [15].…”

“…However, it is not only the natural world that can be inspiration for these different designs; designs can be non-physical, visualisation inspired, human-made or natural (nature inspired) [52]. While many of these designs are implicitly 3D, because they are taken from the natural world (such as ConeTree [53] or hierarchy based visualisation of software [54]), it is clear that the designers do not restrict themselves to keeping a 3D implementation, and inspiration from (say) nature can also be projected into 2D [12].…”

Using Virtual Reality to Facilitate Common Operational Pictures’ Representation

“…The idea is to extend the range of solicited sensory channels and to involve more than just the user's vision. The user is no longer just watching the data from an external point of view: as explained in [17], with Immersive Analytics and VR, the idea is to be surrounded by data that can be touched, heard, or even smelled. The user can move around the data, approaching it from a variety of points of view, using forces and gestures to interact with it.…”

“…This would allow more natural interfaces, matching the usual movements of our bodies. Immersive data offer "opportunities to display substantial quantity of data, and use human metaphors such as front and back (where interesting aspects are in front of us, with those items of less interest pushed behind us)" [17]. Immersion leads to "a demonstrably better perception of datascape geometry, more intuitive data understanding and a better retention of the perceived relationships in the data" [6], and "immersive environments can be exploited to more efficiently extract information from [large amounts of high-dimensional data]" [8].…”