A Structure Design of Virtual and Real Fusion Intelligent Equipment and Multimodal Navigational Interaction Algorithm

Abstract

Virtual experiments have become an interesting research topic in the field of education. However, we found that there are some limitations in the current virtual experiments: first, the researchers used the virtual effects of the simulation to represent the virtual experiments, which led to decrease the immersion of the user's simulated experiments; second, most of the virtual experiments are only mouse or touch screen interactive mode, which reduces the realism of user simulation experiments; third, students independently explore the experimental operation process and spend too much time simulating the experiment, which leads to problems such as overloading the operation and low interaction efficiency. In order to solve the above problems, we propose and implement a multimodal navigational interaction virtual and real fusion chemistry laboratory (MNIVRFCL). We design a new sensing structure intelligent equipment and propose a multimodal navigational interaction algorithm (MMNI) based on auditory and tactile channel, which are verified and applied in MNIVRFCL. The MMNI algorithm can detect user's specific behaviors to understand their behavioral intentions, and then the system guide and rectify users' current correct or incorrect behaviors in the form of voice navigation broadcasts. Finally, we achieve the purpose that students can use virtual and real fusion interactions through tactile and auditory channels, they can independently complete simulations and learning experiments based on experimental navigation. The experimental statistic results show that system's successful understanding of intention is 91.48%, and prove the MNIVRFCL operational load reduce by 23.81% compared to the pure virtual experiment, it can reduce the time consumption and improves the students' interaction efficiency.

Multimodal Information Perception and Understanding: Application of Smart Glove in Virtual‐Reality Fusion Chemistry Experiment Platform

“…In this paper, volunteers were asked to participate in two experiments: (1) the AR experiment of intention acquisition using the MMNI algorithm [ 26 ] and (2) the smart glove chemistry experiment of intention acquisition using the NIAMIU algorithm. Each volunteer must complete concentrated sulfuric acid dilution experiments in the comparison experiment three times, and the intention recognition rate is calculated by recording the number of successful recognitions (successful recognition means that the system prompts the user with the current operation intention by voice, and gives the response results of the operation in real time on the virtual‐reality fusion platform).…”

“…This paper invites 10 volunteers to perform an evaluation of a virtual‐reality fusion experiment, virtual experiment, [ 26 ] NOBOOK [ 1 ] experiment, and real experiment based on smart glove using NASA's TLX table. [ 28 ] We believe that the lower the final scores of mental demands (MD), physical demands (PD), time demands (TD), effort (E), and frustration (F) in the evaluation of the six indicators, the better the effect.…”

“…Furthermore, smart glove is more cognitively capable than traditional data glove. [7][8][9][10][11][12][13] 2) Based on multimodal intent understanding, [14][15][16][17][18][19][20][21][22][23][24][25][26] this paper establishes a navigation interaction algorithm. By integrating information from the user's voice channel and behavior channel, the algorithm can ascertain the user's experimental intent and then guide or rectify them based on their intent.…”

MFA: A Smart Glove with Multimodal Intent Sensing Capability

“…Kadavasal and Oliver [ 10 ] created a virtual-reality driving system for autistic individuals that included physiological signals, brain signals, and eye gaze information, to improve autistic patients' driving abilities. Due to the lack of practical utility and popularity of virtual-reality experimental teaching, Xiao et al [ 11 ] developed a multimodal interaction model that integrates voice and sensor information. Liu et al [ 12 ] proposed a deep learning-based multimodal fusion model that combines three modal data sets: voice commands, hand gestures, and body movements, using various deep neural networks.…”

“…Verification of MFA Algorithm. Although the TMFA [11] uses multichannel data information in the experimental process, its essence is serial fusion of multimodal information, which means that only one channel of information is used in each intent recognition. e essence of MFA is the parallel fusion of multimodal intent probability.…”