The impact of multisensory integration and perceptual load in virtual reality settings on performance, workload and presence

Abstract

Real-world experience is typically multimodal. Evidence indicates that the facilitation in the detection of multisensory stimuli is modulated by the perceptual load, the amount of information involved in the processing of the stimuli. Here, we used a realistic virtual reality environment while concomitantly acquiring Electroencephalography (EEG) and Galvanic Skin Response (GSR) to investigate how multisensory signals impact target detection in two conditions, high and low perceptual load. Different multimodal stimuli (auditory and vibrotactile) were presented, alone or in combination with the visual target. Results showed that only in the high load condition, multisensory stimuli significantly improve performance, compared to visual stimulation alone. Multisensory stimulation also decreases the EEG-based workload. Instead, the perceived workload, according to the “NASA Task Load Index” questionnaire, was reduced only by the trimodal condition (i.e., visual, auditory, tactile). This trimodal stimulation was more effective in enhancing the sense of presence, that is the feeling of being in the virtual environment, compared to the bimodal or unimodal stimulation. Also, we show that in the high load task, the GSR components are higher compared to the low load condition. Finally, the multimodal stimulation (Visual-Audio-Tactile—VAT and Visual-Audio—VA) induced a significant decrease in latency, and a significant increase in the amplitude of the P300 potentials with respect to the unimodal (visual) and visual and tactile bimodal stimulation, suggesting a faster and more effective processing and detection of stimuli if auditory stimulation is included. Overall, these findings provide insights into the relationship between multisensory integration and human behavior and cognition.

The Sample Size Matters: To What Extent the Participant Reduction Affects the Outcomes of a Neuroscientific Research. A Case-Study in Neuromarketing Field

“…In these cases, the lower time resolution would decrease the intra-subject variability, thus impacting the group average of the indexes, and therefore sample size variation could have different effects. Another common neuroscientific experimental approach envisages the analysis of Event-Related Potentials (ERPs) [ 24 , 66 ]; in this case, temporal resolution is even higher (milliseconds), but signal-to-noise ratio is increased by increasing the number of repetitions and computing multiple averages over them. Again, also in this kind of application, sample size variation could have different effects; therefore, neuroscientific field encloses several methodologically different applications.…”

“…In fact, thanks to the increasing shared knowledge and recent scientific evidence produced by the neuroscientific community, behavioural neuroscience is experiencing an exponential growth in terms of applied research [ 13 , 14 , 15 ]. Recently, new terms have been coined, such as Neuroergonomics, Neuromarketing, Neuroaesthetics, and others, to include the new neuroscientific disciplines that aim at investigating neurophysiological correlates of humans’ behaviour at work [ 16 , 17 , 18 ], including driving a car [ 19 , 20 , 21 , 22 , 23 , 24 ], watching television [ 25 , 26 , 27 ], having a sensorial experience [ 28 , 29 , 30 ], and in any other activity of daily life. In the context of applied neurosciences, when physiological signals are acquired in real environments, several factors work against the ease of participant recruitment.…”

“…This number was deemed appropriate to assume the results were reliable, i.e., representative of the theorical population, in line with the scientific literature [32]. These subgroups were chosen since: (i) in the literature there is a large consensus considering [16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32] the optimal range of participants in neuroimaging studies [33]; (ii) in order to perform Analysis of Variance, the number of cases (i.e., 630) has to be much higher than the number of groups (i.e., five). The experimental task consisted of watching four TV spots with different lengths (30,30,20, and 15 s) during which electroencephalographic (EEG) signal and autonomic measures, i.e., heart rate (HR) and galvanic skin response (GSR), were acquired.…”

Efference information flow during skill acquisition mediate its interaction with medical simulation technology

“…Specifically, there is a need to study the directional information flow that creates the sensorimotor mapping during motor skill acquisition. Here, the novice brain builds a multimodal cognitive-perceptual model 3,4 under perceptual load 5 to learn a goal-directed action during skill acquisition 1 . Under the forward model concept 6,7 , the central nervous system internally simulates the motor behavior in planning, control and learning such that the prediction of the sensory input is compared with the reafferent sensory input to determine discrepancy, e.g., at the frontal eye field for the visuosaccadic system 8 .…”

“…FLS is being used to measure and document those skills for medical practitioners where the understanding of the brain-behavior relationship is crucial for informed training and assessment 18 , especially in the context of perception-action coupling 19 in physical versus VR simulators during laparoscopic surgery training 20 . For example, surgeons rely on two-dimensional (2D) visualization of the three-dimensional (3D) surgical field at a reduced depth and tactile perception 21 where 3D vision has been shown to speed up laparoscopic training 22 possibly by reducing perceptual load 5 that is difficult to establish without the investigation of the information flow in the brain. Moreover, VR-driven sensorimotor stimulation may have harmful aftereffects 23 that need investigation of the brainbehavior relationship in the context of sensation weighting in the perception-action cycle 19 .…”