An audiovisual interface-based drumming system for multimodal human–robot interaction

İnce, Gökhan; Yorganci, Rabia; Ozkul, Ahmet Semih; Duman, Taha Berkay; Köse, Hatice

doi:10.1007/s12193-020-00352-w

Cited by 9 publications

(3 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ince et al ( 2021 ) present research on an audiovisual UI-based drumming platform for multimodal HRI, creating an audiovisual communicative interface by combining communicative multimodal drumming with humanoid robots. Chen et al ( 2021 ) explore multimodal emotion identification and intent understanding, presenting various modalities of mental feature extraction and emotion recognition methods, and applying them in practice to achieve HRI.…”

Section: Recent Advancements Of Application For Multi-modal Human–rob...mentioning

confidence: 99%

Recent advancements in multimodal human–robot interaction

Yang

et al. 2023

Front. Neurorobot.

View full text Add to dashboard Cite

Robotics have advanced significantly over the years, and human–robot interaction (HRI) is now playing an important role in delivering the best user experience, cutting down on laborious tasks, and raising public acceptance of robots. New HRI approaches are necessary to promote the evolution of robots, with a more natural and flexible interaction manner clearly the most crucial. As a newly emerging approach to HRI, multimodal HRI is a method for individuals to communicate with a robot using various modalities, including voice, image, text, eye movement, and touch, as well as bio-signals like EEG and ECG. It is a broad field closely related to cognitive science, ergonomics, multimedia technology, and virtual reality, with numerous applications springing up each year. However, little research has been done to summarize the current development and future trend of HRI. To this end, this paper systematically reviews the state of the art of multimodal HRI on its applications by summing up the latest research articles relevant to this field. Moreover, the research development in terms of the input signal and the output signal is also covered in this manuscript.

show abstract

Section: Recent Advancements Of Application For Multi-modal Human–rob...mentioning

confidence: 99%

Recent advancements in multimodal human–robot interaction

Yang

et al. 2023

Front. Neurorobot.

View full text Add to dashboard Cite

show abstract

“…Multi-modal perception, as a method of integrating various sensory information (Wu et al, 2022 ), has also gained increasing attention in the field of robotics. By simultaneously fusing visual, motion, and distance information (Hong et al, 2020 ), multi-modal perception enables robots to have a comprehensive understanding of the environment and tasks, thus improving decision-making accuracy (Ince et al, 2021 ). Nevertheless, research on multi-modal perception in enhancing the shooting skills of basketball robots is still relatively limited.…”

Section: Introductionmentioning

confidence: 99%

Research on deep reinforcement learning basketball robot shooting skills improvement based on end to end architecture and multi-modal perception

Zhang,

Tao

2023

Front. Neurorobot.

View full text Add to dashboard Cite

IntroductionIn the realm of basketball, refining shooting skills and decision-making levels using intelligent agents has garnered significant interest. This study addresses the challenge by introducing an innovative framework that combines multi-modal perception and deep reinforcement learning. The goal is to create basketball robots capable of executing precise shots and informed choices by effectively integrating sensory inputs and learned strategies.MethodsThe proposed approach consists of three main components: multi-modal perception, deep reinforcement learning, and end-to-end architecture. Multi-modal perception leverages the multi-head attention mechanism (MATT) to merge visual, motion, and distance cues for a holistic perception of the basketball scenario. The deep reinforcement learning framework utilizes the Deep Q-Network (DQN) algorithm, enabling the robots to learn optimal shooting strategies over iterative interactions with the environment. The end-to-end architecture connects these components, allowing seamless integration of perception and decision-making processes.ResultsThe experiments conducted demonstrate the effectiveness of the proposed approach. Basketball robots equipped with multi-modal perception and deep reinforcement learning exhibit improved shooting accuracy and enhanced decision-making abilities. The multi-head attention mechanism enhances the robots' perception of complex scenes, leading to more accurate shooting decisions. The application of the DQN algorithm results in gradual skill improvement and strategic optimization through interaction with the environment.DiscussionThe integration of multi-modal perception and deep reinforcement learning within an end-to-end architecture presents a promising avenue for advancing basketball robot training and performance. The ability to fuse diverse sensory inputs and learned strategies empowers robots to make informed decisions and execute accurate shots. The research not only contributes to the field of robotics but also has potential implications for human basketball training and coaching methodologies.

show abstract

“…In terms of intelligent hardware, the mainstream method in the industry is processing signals by microphone arrays and eliminating noise by hardware. However, when the environment is complex and noisy, there is still a large bottleneck in speech recognition [5][6][7]. The next generation of revolutionary HCI technology may not impact the whole industry, such as the emergence of graphical interfaces and touch technology, but may use data-driven intelligence to realize the potential revolution of HCI [8].…”

Section: Introductionmentioning

confidence: 99%

Deep Learning for Intelligent Human–Computer Interaction

Poiesi

Dong

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

In recent years, gesture recognition and speech recognition, as important input methods in Human–Computer Interaction (HCI), have been widely used in the field of virtual reality. In particular, with the rapid development of deep learning, artificial intelligence, and other computer technologies, gesture recognition and speech recognition have achieved breakthrough research progress. The search platform used in this work is mainly the Google Academic and literature database Web of Science. According to the keywords related to HCI and deep learning, such as “intelligent HCI”, “speech recognition”, “gesture recognition”, and “natural language processing”, nearly 1000 studies were selected. Then, nearly 500 studies of research methods were selected and 100 studies were finally selected as the research content of this work after five years (2019–2022) of year screening. First, the current situation of the HCI intelligent system is analyzed, the realization of gesture interaction and voice interaction in HCI is summarized, and the advantages brought by deep learning are selected for research. Then, the core concepts of gesture interaction are introduced and the progress of gesture recognition and speech recognition interaction is analyzed. Furthermore, the representative applications of gesture recognition and speech recognition interaction are described. Finally, the current HCI in the direction of natural language processing is investigated. The results show that the combination of intelligent HCI and deep learning is deeply applied in gesture recognition, speech recognition, emotion recognition, and intelligent robot direction. A wide variety of recognition methods were proposed in related research fields and verified by experiments. Compared with interactive methods without deep learning, high recognition accuracy was achieved. In Human–Machine Interfaces (HMIs) with voice support, context plays an important role in improving user interfaces. Whether it is voice search, mobile communication, or children’s speech recognition, HCI combined with deep learning can maintain better robustness. The combination of convolutional neural networks and long short-term memory networks can greatly improve the accuracy and precision of action recognition. Therefore, in the future, the application field of HCI will involve more industries and greater prospects are expected.

show abstract

An audiovisual interface-based drumming system for multimodal human–robot interaction

Cited by 9 publications

References 41 publications

Recent advancements in multimodal human–robot interaction

Recent advancements in multimodal human–robot interaction

Research on deep reinforcement learning basketball robot shooting skills improvement based on end to end architecture and multi-modal perception

Deep Learning for Intelligent Human–Computer Interaction

Contact Info

Product

Resources

About