From sound to shape: Auditory perception of drawing movements.

Abstract

This study investigates the human ability to perceive biological movements through friction sounds produced by drawings and, furthermore, the ability to recover drawn shapes from the friction sounds generated. In a first experiment, friction sounds, real-time synthesized and modulated by the velocity profile of the drawing gesture, revealed that subjects associated a biological movement to those sounds whose timbre variations were generated by velocity profiles following the 1/3 power law. This finding demonstrates that sounds can adequately inform about human movements if their acoustic characteristics are in accordance with the kinematic rule governing actual movements. Further investigations of our ability to recognize drawn shapes were carried out in 2 association tasks in which both recorded and synthesized sounds had to be associated to both distinct and similar visual shapes. Results revealed that, for both synthesized and recorded sounds, subjects made correct associations for distinct shapes, although some confusion was observed for similar shapes. The comparisons made between recorded and synthesized sounds lead to conclude that the timbre variations induced by the velocity profile enabled the shape recognition. The results are discussed in the context of the ecological and ideomotor frameworks.

Expert players accurately detect an opponent’s movement intentions through sound alone.

“…As highlighted in previous studies, it is through the structural and transformational invariants present in the sound (Thoret, Aramaki, Kronland-Martinet, Velay, & Ystad, 2014), that participants can anticipate the attacker's movement kinematics (e.g. velocity, trajectory, position).…”

The effect of real-time auditory feedback on learning new characters

“…This step was recently performed by Thoret, Aramaki, Kronland-Martinet, Velay, and Ystad (2014) who demonstrated that adult subjects were able to infer by ear what was drawn based, only, on the generated friction sounds. Their rationale was that, although handwriting is not really a noisy activity, the slight sounds generated by the pen friction on the paper can be heard in a silent environment.…”

Drawing learning support system with the past brilliant learner-self using visual and auditory feedback

“…For example, some studies have shown providing sound synthesis can improve hands gesture accuracy on mobile devices [4], communication with geometrical data on screen [1], and impressions during the writing task [3]. Futhermore, some recent studies suggest audio feedback is effective for the drawing shape recognition [2,5]. We aim to apply visual and auditory effects for improvement of drawing skill.…”

“…One is to apply an existed friction model with changing sound according to velocity and pressure of pen tip. This is because Thoret et al [5] suggest the possibility of the relation between drawn shapes and friction sounds, which strongly suggests that common rules constrain perception and action of biological motions. The other one is to synthesize sound with changing the audio parameters according to the drawing motion (e.g., x axis: panning, y axis: frequency, pressure: volume) and to provide a metronome sound.…”