Crossmodal matching

Abstract

Multimodal displays, i.e., displays that distribute information across multiple sensory channels (mainly vision, hearing, and touch), have received considerable attention in recent years. To be effective, their design needs to be based on a firm understanding of how information is processed both within and across modalities. However, most studies on crossmodal information processing, to date, suffer from a methodological shortcoming: they fail to perform crossmodal matching to ensure that modality is not confounded with other stimulus properties, such as salience. One reason for this shortcoming is the fact that there is no agreed-upon crossmodal matching technique, and that existing approaches suffer from limitations. The goal of the present study is to develop and validate a more reliable crossmodal matching method that employs repeated matching. To this end, six participants were asked to use this technique and match a series of 54 modality pairings involving vision, audition, and touch. Results show that the intraindividual variability of participants' matches was significantly less than observed in an earlier technique that involved bidirectional matching and visual feedback. The findings from this research confirm the need for improved crossmodal matching procedures and for employing them in advance of conducting experiments on multisensory information processing.

Crossmodal matching

“…The results showed significant within and between subject variability and confirm findings from previous studies (Pitts et al, 2013(Pitts et al, , 2015Pitts & Sarter, 2014). The findings of high variability of matches between different modalities support the need for crossmodal matching prior to multimodal studies, especially with regards to those comparing performance across modalities and evaluating the effectiveness of multimodal displays.…”

“…Other participants described that their strategy was recalling previous matches in making their selections. The reliance on visual feedback has also been noted in previous work and it has been recommended that visual feedback should not be used (Pitts & Sarter, 2014).…”

“…Since there is no consensus on a crossmodal matching method for multimodal research, this points to a need of a reliable technique. Pitts, Lu, & Sarter (2013) have conducted research supporting this goal by developing and testing a method similar to Stevens' method of adjustment -the participant was able to directly control the intensity of the to be matched modality via a sliding scale (Pitts, Riggs, & Sarter, 2015;Pitts & Sarter, 2014). A reliable crossmodal matching method should minimize or be able to account for within and between-subject variability.…”

“…Since both methods have been established and validated by Stevens, the present work evaluated a new crossmodal matching method based on Stevens' method of bracketing and compared it to Stevens' method of adjustment. Previous crossmodal matching studies have found significant within and between-subject variability in matching the intensity of a stimulus in one modality to another modality (Pitts et al, 2013;Pitts & Sarter, 2014). In order to develop an effective crossmodal matching method, which accounts for subject variability, the sources of variability need to be identified.…”

Comparison of vibrotactile and joint-torque feedback in a myoelectric upper-limb prosthesis

“…Prior to performing the experimental task, we performed a cross-modal matching task based on the paradigm developed by Pitts et al to calibrate the two haptic modalities, vibrotactile and joint-torque feedback, in terms of their stimulus intensity [22]. This step is done so that any difference in experimental performance between the two modalities is the result of differences in the sensory encoding and not differences in the perceived stimulus intensity.…”