Multisensory Analytics: Case of Visual-auditory Analysis of Scalar Fields

Abstract

A well-known definition of visualization is the mapping of initial data to a visual representation, which can be perceived and interpreted by humans. Human senses include not only vision, but also hearing, sense of touch, smell and others including their combinations. Visual analytics and its more general version that we call Multisensory Analytics are areas that consider visualization as one of its components. We present a particular case of the multisensory analytics with a hybrid visual-auditory representation of data to show how auditory display can be used in the context of data analysis. Some generalizations based on using real-valued vector functions for solving data analysis problems by means of multisensory analytics are proposed. These generalizations might be considered as a first step to formalization of the correspondence between the initial data and various sensory stimuli. An illustration of our approach with a case study of analysis of a scalar field using both visual and auditory data representations is given. 322Malikova E.

Multisensory Analytics for Several Interconnected Scalar Fields Analysis

“…The auditory perception brings the unique advantage to distinguish even small variations in the parameters of the single sound wave and to compare sound waves and thus can efficiently complement standard visual analyses, and help solving perceptional issues it often has to face [7], [8]. As it was already noticed in [9] currently, it is considered that any person may be trained to develop an ear for music and ability to adequately judge on all its nuances allows one to take advantage of the most advanced extended sound analysis capabilities as well. In [10] the procedures of time-varying data representation in the graphical form using a musical accompaniment are considered.…”

“…We'll give detailed examples and explain analyses process in details. As previously [9], we propose some generalizations on base multimedia coordinates approaches for specific type of multidimensional data multisensory analysis -scalar fields, bringing together some most well known interactive, photometric and geometrical techniques and demonstrating how they can be extended by other multisensory techniques on the example of sound.…”

“…In [9] we demonstrated basic interactive techniques, when sound with predefined by data characteristics is produced in selected point. In this article we are concentrating on more extended that in general may be used by researches without preliminary theoretical knowledge in music analyses, although these knowledge will make analyses more efficient and extended, and on base of them analyses may be carried out even without visual information.…”

“…Then corresponding visual, sound and other stimuli can be generated using some specialized rendering procedures for further multicensory analysis. Based on the visual analytics process as presented in [17] and the idea of an intermediate multidimensional geometric representation of initial data [18], we proposed [9] the following interpretation of the basic multisensory analysis process 1. I from initial data D to insight I, which is the goal of the entire process.…”

“…To obtain a multisensory representation we need to create a spatial scene [9], which is an assembly of spatial objects with their geometric, optical, sound and others descriptions. Then corresponding visual, sound and other stimuli can be generated using some specialized rendering procedures for further multicensory analysis.…”