Latent Semantic Analysis for Multimodal User Input With Speech and Gestures

Hui, Pui-Yu; Meng, Helen

doi:10.1109/taslp.2013.2294586

Cited by 15 publications

(6 citation statements)

References 36 publications

(38 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Как правило, потоки неструктурированных данных, генерируемые такой подсистемой, настолько интенсивны, что производительности современных систем интеллектуального принятия решений недостаточно для онтологизации всех объектов, событий и ситуаций, информацию о которых несут эти потоки, в режиме реального времени [11,12].…”

Section: рис 2 мультиакторная архитектура агнейрона в программе имита...unclassified

Multi-agent neurocognitive algorithm for controlling the reference of speech events of communication of a general artificial intelligence agent in a situation of synchronous multiple dialogues

Nagoev

2023

News of KBSC RAS

View full text Add to dashboard Cite

Кабардино-Балкарский научный центр Российской академии наук 360010, Россия, г. Нальчик, ул. Балкарова, 2Аннотация. Разработаны основные принципы, модели и алгоритмы управления референцией речевых сообщений на основе создания двухконтурной модели мультиагентной нейрокогнитивной архитектуры -суперинтеллектона, реализующего взаимодействие интеллектона подсознания и интеллектона сознания. Сформированы требования к онтологиям агента общего искусственного интеллекта, условия их формирования и обозначены функциональные узлы нейрокогнитивных архитектур, необходимые для их эффективного формирования в режиме обучения. Полученные результаты могут быть применены при создании систем распознавания и понимания речи, работоспособных при применении в зашумленных средах и ситуациях множественных синхронных диалогов для повышения качества распознавания с использованием понимания контекста ситуаций.

show abstract

Section: рис 2 мультиакторная архитектура агнейрона в программе имита...unclassified

Multi-agent neurocognitive algorithm for controlling the reference of speech events of communication of a general artificial intelligence agent in a situation of synchronous multiple dialogues

Nagoev

2023

News of KBSC RAS

View full text Add to dashboard Cite

show abstract

“…Minotto et al [35] used an RGB camera and depth sensor as input stream and proposed a multimodal speaker diarization algorithm to extract speech features for fusion. Hui et al [36] analyzed and fused multimodal languages of speech and gestures based on latent semantic analysis (LSA).…”

Section: B Multimodal Interactionmentioning

confidence: 99%

A Structure Design of Virtual and Real Fusion Intelligent Equipment and Multimodal Navigational Interaction Algorithm

et al. 2020

View full text Add to dashboard Cite

Virtual experiments have become an interesting research topic in the field of education. However, we found that there are some limitations in the current virtual experiments: first, the researchers used the virtual effects of the simulation to represent the virtual experiments, which led to decrease the immersion of the user's simulated experiments; second, most of the virtual experiments are only mouse or touch screen interactive mode, which reduces the realism of user simulation experiments; third, students independently explore the experimental operation process and spend too much time simulating the experiment, which leads to problems such as overloading the operation and low interaction efficiency. In order to solve the above problems, we propose and implement a multimodal navigational interaction virtual and real fusion chemistry laboratory (MNIVRFCL). We design a new sensing structure intelligent equipment and propose a multimodal navigational interaction algorithm (MMNI) based on auditory and tactile channel, which are verified and applied in MNIVRFCL. The MMNI algorithm can detect user's specific behaviors to understand their behavioral intentions, and then the system guide and rectify users' current correct or incorrect behaviors in the form of voice navigation broadcasts. Finally, we achieve the purpose that students can use virtual and real fusion interactions through tactile and auditory channels, they can independently complete simulations and learning experiments based on experimental navigation. The experimental statistic results show that system's successful understanding of intention is 91.48%, and prove the MNIVRFCL operational load reduce by 23.81% compared to the pure virtual experiment, it can reduce the time consumption and improves the students' interaction efficiency.

show abstract

“…Their experiments showed that multimodality was better than single mode recognition. In 2014, Hui and Meng [29] fused the user's voice and pen at the feature layer. The experimental results showed that the user's intention was understood and that the robustness was improved.…”

Section: Related Workmentioning

confidence: 99%

Research on Multimodal Perceptual Navigational Virtual and Real Fusion Intelligent Experiment Equipment and Algorithm

et al. 2020

View full text Add to dashboard Cite

Virtual experiment is an important field of human-computer interaction. With more and more virtual laboratories emerging, we found that problems regarding virtual experiments are rising. Such problems can be listed as follows: First, human-computer interaction has lower efficiency during the process of virtual experiment, which means the computer cannot understand the user's intention thus leading to incorrect operation. Second, there are less detections for false behavior during experiments. Third, the virtual laboratory's sense of operation and realism is not strong. In order to solve the above problems, the multimodal sensing navigation virtual and real fusion laboratory (MSNVRFL) was designed and implemented in this paper. We design a new set of experimental equipment with the function of cognition and study a multimodal fusion model and algorithm for chemical experiments, which are both finally verified and applied in MSNVRFL. By using multimodal fusion perception algorithm, the user's true intentions can be understood and the human-computer interaction efficiency can be improved. By carrying out a virtual experiment with the mold of virtual and real fusion, problems like resources wasting and dangers happened during experiment can be avoided, user's sense of operation and realism can be improved. In addition, teaching navigation and wrong operation behavior reminders are provided for users. The experimental result shows that our method can improve the efficiency of human-computer interaction, reduce the user's cognitive load, strengthen the user's sense of reality and operation and stimulate students' interest in learning. INDEX TERMS Multimodal fusion, virtual experiments, intelligent teaching, human-computer interaction.

show abstract

Latent Semantic Analysis for Multimodal User Input With Speech and Gestures

Cited by 15 publications

References 36 publications

Multi-agent neurocognitive algorithm for controlling the reference of speech events of communication of a general artificial intelligence agent in a situation of synchronous multiple dialogues

Multi-agent neurocognitive algorithm for controlling the reference of speech events of communication of a general artificial intelligence agent in a situation of synchronous multiple dialogues

A Structure Design of Virtual and Real Fusion Intelligent Equipment and Multimodal Navigational Interaction Algorithm

Research on Multimodal Perceptual Navigational Virtual and Real Fusion Intelligent Experiment Equipment and Algorithm

Contact Info

Product

Resources

About