Research on Intelligent Multimodal Gesture-Guided Control of Robotic Arms

Abstract

With the continuous advancement of industrial intelligence, robotic arms have become essential tools for sorting materials and addressing safety hazards in work environments. Traditional robotic arm control technologies, while playing a significant role in high-risk environments such as factories, nuclear power plants, and toxic substance handling sites, are limited by the complexity of their operation and programming, as well as the high cost of wearable equipment. This research proposes a multimodal robotic arm control scheme that does not require the use of wearable sensors, enhancing the operational flexibility and safety of robotic arms in tasks such as material sorting, quality inspection, and hazard elimination. The proposed scheme captures hand gestures through computer vision technology and utilizes artificial intelligence to parse natural language commands, ultimately integrating multimodal signals to achieve precise control of the robotic arm. Experimental results demonstrate that the proposed control scheme can accurately map hand gestures to the robotic arm's coordinate system, efficiently performing sorting and hazard elimination tasks without the need for any wearable sensor devices. This approach has the potential to drive the manufacturing industry towards greater efficiency and intelligence.