Hand gesture-based interactive puppetry system to assist storytelling for children

Abstract

Digital techniques have been used to assist narrative and storytelling, especially in many pedagogical practices. With the rapid development of HCI techniques, saturated with digital media in their daily lives, young children, demands more interactive learning methods and meaningful immersive learning experiences. In this paper, we propose a novel hand gesture-based puppetry storytelling system which provides a more intuitive and natural human computer interaction method for young children to develop narrative ability in virtual story world. Depth motion sensing and hand gestures control technology is utilized in the implementation of user-friendly interaction. Young players could intuitively use hand gestures to manipulate virtual puppet to perform story and interact with different items in virtual environment to assist narration. Based on the result of the evaluation, this novel digital storytelling system shows positive pedagogical functions on children's narrating ability as well as the competencies of cognitive and motor coordination. The usability of the system is preliminary examined in our test, and the results which showed that young children can benefit from playing with Puppet Narrator.

Storytelling Menggunakan Boneka Tangan Oleh Guru: Dapatkah Meningkatkan Keterampilan Regulasi Emosi Anak Tk A?

“…Kedekatan yang ditimbulkan antara guru dan anak berdasarkan data kualitatif merupakan salah satu keunggulan dari storytelling menggunakan boneka tangan. Hal ini disebabkan guru tidak perlu terpaku dengan buku atau tulisan sehingga dapat melakukan improvisasi dan anak menikmati jalannya cerita bersama teman-temannya (Liang et al, 2017). Meski demikian, guru terlihat kurang maksimal dalam memberikan stimulasi regulasi emosi kepada anak.…”

Hand features extractor using hand contour – a case study

“…Based on their methods, hand gesture recognition technology is mainly divided into two categories: data gloves-based method and computer vision-based method. The computer vision-based method can be classified into two main classes [2]: static gestures [3][4][5][6][7] that are done with a hand or both hands with no movement, and dynamic gestures [8,9,2,10,11,12] that are done with a sequence of hand images following a path or a predefined behaviour. Both static and dynamic gestures are extracted from a sequence of hand images (video), but in the first, the gesture has to be the same and to be in the same position, in the second one, gesture has to be a little different in the form or in the position.…”

“…There are more specific hand-related devices like the Leap Motion Controller [12][13][14] that detect mainly finger movements; however, the main drawback with this gadget is when hands are overlapping each other. Also, some applications for hand gestures are proposed in medical environments [15][16][17], in serious games [13] or in pedagogical practices [10]. It is worth mentioning works that use a mix of Kinect, Leap Motion and Oculus Rift [18,19].…”

Two Hand Gesture Based 3D Navigation in Virtual Environments

“…Gestures proposed by Batista et al[112].Liang et al[113] presented a system using Leap Motion controller for the detection of hand gestures. There were different types of modules for navigation such as single hand gestures, designed for children which causes avatar in VE to fly left, right, up, and down as shown inFig.…”