Cross-modal matching

Abstract

Research in the area of multimodal displays and information processing has reported several benefits of distributing information across multiple sensory channels (vision, audition, and touch, in particular). However, with few exceptions, studies on multimodal information processing involve the potential risk of confounding modality with other factors, such as salience, because no cross-modal matching is being performed prior to experiments. To date, no agreed-upon cross-modal matching method has been developed. The goal of our research is to develop and compare the feasibility and validity of various approaches. In this paper, we present the findings for one particular technique that employs cue adjustments and bidirectional matches. Six participants were asked to perform a series of 216 matching tasks for combinations of cues in vision, audition and touch. The results show that participants' matches differed from one another, were inconsistent across trials, and were also a function of the intensity level of the initial cue. The findings from this research further highlight the need for careful matching of multimodal cues in research on multisensory information processing and will result in refinements of the proposed technique.

Crossmodal matching

“…The results showed significant within and between subject variability and confirm findings from previous studies (Pitts et al, 2013(Pitts et al, , 2015Pitts & Sarter, 2014). The findings of high variability of matches between different modalities support the need for crossmodal matching prior to multimodal studies, especially with regards to those comparing performance across modalities and evaluating the effectiveness of multimodal displays.…”

“…However, post hoc tests showed that significance occurred only at the lowest reference cue intensity. This may be due to the fact that participants tend to overestimate lower values supporting findings from previous studies (Pitts et al, 2013). Another explanation for the differences in match values between the interface types may be due to the presence of visual feedback.…”

“…Since there is no consensus on a crossmodal matching method for multimodal research, this points to a need of a reliable technique. Pitts, Lu, & Sarter (2013) have conducted research supporting this goal by developing and testing a method similar to Stevens' method of adjustment -the participant was able to directly control the intensity of the to be matched modality via a sliding scale (Pitts, Riggs, & Sarter, 2015;Pitts & Sarter, 2014). A reliable crossmodal matching method should minimize or be able to account for within and between-subject variability.…”

“…A reliable crossmodal matching method should minimize or be able to account for within and between-subject variability. The work by Pitts et al (2013Pitts et al ( , 2014Pitts et al ( , 2015 provides a good foundation in developing a new crossmodal matching method based on Stevens' method of adjustment, but has not considered the method of bracketing as an alternative. Since both methods have been established and validated by Stevens, the present work evaluated a new crossmodal matching method based on Stevens' method of bracketing and compared it to Stevens' method of adjustment.…”

“…Since both methods have been established and validated by Stevens, the present work evaluated a new crossmodal matching method based on Stevens' method of bracketing and compared it to Stevens' method of adjustment. Previous crossmodal matching studies have found significant within and between-subject variability in matching the intensity of a stimulus in one modality to another modality (Pitts et al, 2013;Pitts & Sarter, 2014). In order to develop an effective crossmodal matching method, which accounts for subject variability, the sources of variability need to be identified.…”