Interactive sonification of images in serious games as an education aid for visually impaired children

Abstract

The paper presents an application for interactive sonification of images intended for use on mobile devices in education of blind children at elementary school level. The paper proposes novel sonification algorithms for converting colour and grayscale images into sound. The blind user can interactively explore image content through multi‐touch gestures and select image sub‐regions for sonification with real‐time control of the synthesized sound parameters such as amplitude, frequency and timbre. Additionally, images may contain text fields read by text‐to‐speech synthesizer. The usability of one of the proposed sonification schemes is tested by collecting data on the tracking accuracy and recognition speed of basic shapes, such as lines, curves, as well as figures and simple functions. In order to facilitate the learning process for blind children, a number of games were proposed that use the devised sonification schemes. The first game—“Hear the Invisible”—is intended for two players—one child draws a shape and the task of the other one is to guess the displayed shape by means of the available sonification methods. The second proposed game is “Follow the Rabbit” and is intended for a single player who tracks a colourful “Rabbit” that runs along a path representing a given geometric shape. The obtained results show that the proposed sonification methods allow to reach previously unattainable levels of flexibility in exploration of shapes and colours. The main application of the described interactive tool is to teach geometry and mathematics in schools for blind children. The developed games are meant to enhance the learning process and motivate the children. What is known about the topic Sonification is a generic term for data‐based non‐speech sound generation. Interactive sonification is a relatively fresh area of study, where the listener heavily influences the process of sound synthesis. Earlier studies on sonification and educational computer games have shown that it is possible to demonstrate shapes such as plots and geometry using only non‐verbal sounds. What the paper adds The paper proposes an interactive sonification algorithm that can be used on greyscale or colour images using additive synthesis and the HSV colour representation. The presented software also allows to record and analyse user interaction with the images, making it a good research tool for current and future sonification algorithms. The paper presents a pilot study with blind children using two games that improve the learning process of the proposed algorithms. Implications for practice and/or policy Once the software is openly released it can be easily installed on any Android OS device to allow blind users to analyse images by touch. The proposed games train the users the basics of the sonification algorithm in a competitive context. The software can be especially useful in education of blind children, eg, in teaching geometry, math and in training visual and spatial imagination.

A content analysis of research on technology use for teaching mathematics to students with disabilities: word networks and topic modeling

“…Topics such as auditory support and support for visual materials were often investigated for students with visual impairments. The highly associated words for these topics were matched with representative studies: an interactive sonification of images in serious games as auditory support (Radecki et al, 2020 ) and textual materials (words, math expressions) describing visual images as support for visual materials (Emerson & Anderson, 2018 ).…”

Game Animasi Animal Karambol Berbasis Aplikasi Android untuk Meningkatkan Pengetahuan Sains pada Anak

“…Desain strategi pemanfaatan game edukasi biasa dilakukan dengan dua cara, yaitu memainkan game dalam pengaturan pembelajaran atau belajar langsung melalui game (Hartatik, 2019;Hartatik et al, 2019;Utesch et al, 2020;Yunanto et al, 2019). Game edukasi merupakan bentuk permainan yang dirancang untuk membantu pebelajar mencapai tujuan pembelajaran tertentu dan sekaligus memberikan motivasi (Denni et al, 2021;Kristianti et al, 2018;Radecki et al, 2020;Robertson et al, 2018;Wirawan et al, 2013).…”

Developing Dynamic Audio Navigation UIs to Pinpoint Elements in Tactile Graphics

“…The most common method for BVI people to locate graphic elements in touch screens is the trial and error strategy. Users explore visual elements individually with touch by requesting additional audio information [9,12] or receiving vibration feedback [13]. However, BVI pointed out that a user interface to pinpoint elements in touch screens would be helpful [9].…”