Hand Gestures Replicating Robot Arm based on MediaPipe

Abstract

A robotic arm is any variety of programmable mechanical devices designed to operate items like a human arm and is one of the most beneficial innovations of the 20th century, quickly becoming a cornerstone of many industries. It can perform a variety of tasks and duties that may be time-consuming, difficult, or dangerous to humans. The gesture-based control interface offers many opportunities for more natural, configurable, and easy human-machine interaction. It can expand the capabilities of the GUI and command line interfaces that we use today with the mouse and keyboard. This work proposed changing the concept of remote controls for operating a hand-operated robotic arm to get rid of buttons and joysticks by replacing them with a more intuitive approach to controlling a robotic arm via the hand gestures of the user. The proposed system performs vision-based hand gesture recognition and a robot arm that can replicate the user's hand gestures using image processing. The system detects and recognizes hand gestures using Python and sends a command to the microcontroller which is the Arduino board connected to the robot arm to replicate the recognized gesture. Five servo motors are connected to the Arduino Nano to control the fingers of the robot arm; These servos are related to the robot arm prototype. It is worth noting that this system was able to repeat the user's hand gestures with an accuracy of up to 96%.