Development of a Learning-aid tool using Hand Gesture Based Human Computer Interaction System

Boruah, Bidyut Jyoti; Talukdar, Anjan Kumar; Sarma, Kandarpa Kumar

doi:10.1109/acts53447.2021.9708354

Cited by 9 publications

(4 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The open-source MediaPipe framework [56] was used for object detection and tracking in conjunction with an algorithm developed for the present study. For the preprocessing of the data, the SMOTE algorithm [57] was implemented. Furthermore, the UGV used in this study was equipped with a variety of sensors including (i) the depth camera mentioned above, (ii) a 3D laser scanner (Velodyne Lidar Inc., San Jose, CA, USA), (iii) an RTK GPS (S850 GNSS Receiver, Stonex Inc., Concord, NH, USA), and (iv) IMUs (UM7 IMU, RedShift Labs, Studfield, Victoria, Australia).…”

Section: Discussion and Main Conclusionmentioning

confidence: 99%

“…The predefined hand gestures, which are depicted in Figure 1a-e, are commonly known as "fist", "okay", "victory", "flat", and "rock". For the purpose of identifying the position of hands, the MediaPipe framework was implemented [54], similarly to recent studies such as [55][56][57]. MediaPipe is an open-source framework created by Google that offers various ML solutions for object detection and tracking.…”

Section: Data Acquisitionmentioning

confidence: 99%

See 1 more Smart Citation

An Integrated Real-Time Hand Gesture Recognition Framework for Human–Robot Interaction in Agriculture

et al. 2022

View full text Add to dashboard Cite

Incorporating hand gesture recognition in human–robot interaction has the potential to provide a natural way of communication, thus contributing to a more fluid collaboration toward optimizing the efficiency of the application at hand and overcoming possible challenges. A very promising field of interest is agriculture, owing to its complex and dynamic environments. The aim of this study was twofold: (a) to develop a real-time skeleton-based recognition system for five hand gestures using a depth camera and machine learning, and (b) to enable a real-time human–robot interaction framework and test it in different scenarios. For this purpose, six machine learning classifiers were tested, while the Robot Operating System (ROS) software was utilized for “translating” the gestures into five commands to be executed by the robot. Furthermore, the developed system was successfully tested in outdoor experimental sessions that included either one or two persons. In the last case, the robot, based on the recognized gesture, could distinguish which of the two workers required help, follow the “locked” person, stop, return to a target location, or “unlock” them. For the sake of safety, the robot navigated with a preset socially accepted speed while keeping a safe distance in all interactions.

show abstract

Section: Discussion and Main Conclusionmentioning

confidence: 99%

Section: Data Acquisitionmentioning

confidence: 99%

An Integrated Real-Time Hand Gesture Recognition Framework for Human–Robot Interaction in Agriculture

et al. 2022

View full text Add to dashboard Cite

show abstract

“…al. [14], used the machine learning specifically uses Media Pipe to detect hand gesture to control computer mouse through these hand gestures. In MP, Hand Gestures were detected by two faces.…”

Section: Related Workmentioning

confidence: 99%

“…al. [14], Subhangi Adhikary et.al [15], A. Halder et.al [16], Prakash et.al. [17], Ali Suryaperdana Agoes et.…”

Section: Actuators Power Consumption Testmentioning

confidence: 99%

Tracked Robot Control with Hand Gesture Based on MediaPipe

Wameed,

M. ALKAMACHI,

Erçelebi

2023

alkej

View full text Add to dashboard Cite

Hand gestures are currently considered one of the most accurate ways to communicate in many applications, such as sign language, controlling robots, the virtual world, smart homes, and the field of video games. Several techniques are used to detect and classify hand gestures, for instance using gloves that contain several sensors or depending on computer vision. In this work, computer vision is utilized instead of using gloves to control the robot's movement. That is because gloves need complicated electrical connections that limit user mobility, sensors may be costly to replace, and gloves can spread skin illnesses between users. Based on computer vision, the MediaPipe (MP) method is used. This method is a modern method that is discovered by Google. This method is described by detecting and classifying hand gestures by identifying 21 three-dimensional points on the hand, and by comparing the dimensions of those points. This is how the hand gestures are classified. After detecting and classifying the hand gestures, the system controls the tracked robot through hand gestures in real time, as each hand gesture has a specific movement that the tracked robot performs. In this work, some important paragraphs concluded that the MP method is more accurate and faster in response than the Deep Learning (DL) method, specifically the Convolution Neural Network (CNN). The experimental results shows the accuracy of this method in real time through the effect of environmental elements decreases in some cases when environmental factors change. Environmental elements are such light intensity, distance, and tilt angle (between the hand gesture and camera).The reason for this is that in some cases, the fingers are closed together, and some fingers are not fully closed or opened and the accuracy of the camera used is not good with the changing environmental factors. This leads to the inability of the algorithm used to classify hand gestures correctly (the classification accuracy decrease), and thus response time of the tracked robot's movement increases. That does not present possibility for the system to determine whether the finger is closed or opened.

show abstract