“Let Me Hear Your Handwriting!” Evaluating the Movement Fluency from Its Sonification

Abstract

The quality of handwriting is evaluated from the visual inspection of its legibility and not from the movement that generates the trace. Although handwriting is achieved in silence, adding sounds to handwriting movement might help towards its perception, provided that these sounds are meaningful. This study evaluated the ability to judge handwriting quality from the auditory perception of the underlying sonified movement, without seeing the written trace. In a first experiment, samples of a word written by children with dysgraphia, proficient children writers, and proficient adult writers were collected with a graphic tablet. Then, the pen velocity, the fluency, and the axial pen pressure were sonified in order to create forty-five audio files. In a second experiment, these files were presented to 48 adult listeners who had to mark the underlying unseen handwriting. In order to evaluate the relevance of the sonification strategy, two experimental conditions were compared. In a first ‘implicit’ condition, the listeners made their judgment without any knowledge of the mapping between the sounds and the handwriting variables. In a second ‘explicit’ condition, they knew what the sonified variables corresponded to and the evaluation criteria. Results showed that, under the implicit condition, two thirds of the listeners marked the three groups of writers differently. In the explicit condition, all listeners marked the dysgraphic handwriting lower than that of the two other groups. In a third experiment, the scores given from the auditory evaluation were compared to the scores given by 16 other adults from the visual evaluation of the trace. Results revealed that auditory evaluation was more relevant than the visual evaluation for evaluating a dysgraphic handwriting. Handwriting sonification might therefore be a relevant tool allowing a therapist to complete the visual assessment of the written trace by an auditory control of the handwriting movement quality.

Seeing Circles and Drawing Ellipses: When Sound Biases Reproduction of Visual Motion

“…linking the properties of drawing movements to their evocation through timbre variations of friction sounds. Studies by Danna et al [ 33 ] confirmed such binding between friction sounds and drawing movements by showing that the quality of handwriting can be accurately judged through friction sounds evoking the movement velocity. Such unified percepts that link sounds and actions have also been shown for footstep sounds [ 34 ].…”

Interactive sonification of a fluid dance movement: an exploratory study

“…The sound model used for sonification was created using the SuperCollider 7 programming language. The model exploited harmonic pitch sensitivity, physical interpretation of spectral slope and amplitude, as well as a noise generator that could move from coarse, gravel-like sounds, to smoother sounds reminiscent of a water stream or wind.…”

“…-Slider 2: The amount of high frequency content in the noise source. 7 http://supercollider.github.io/.…”

“…3 In previous studies on sonification of continuous body gestures, researchers have identified certain sound properties that are associated with fluent or irregular movements. For example, in a study on the sonification of handwriting it was found that sound models characterized by low frequency components were more suitable for both aiding and communicating fluency of movements, while sound models characterized by high frequency crackling sounds (impact sounds) were suitable for portraying jerky hand movements lacking fluency [7]. In another study focusing on the use of sounds in learning of movement kinematics, researchers found that sounds that were noisier or had loud high-frequency components could help users identify motion behavior [2].…”

Timbre from Sound Synthesis and High-Level Control Perspectives

“…When associated with a biological law, a specific calibration of the velocity profile enables the evocation of a human gesture (Thoret et al 2014). Such a synthesis tool has been used in several applications, for instance for video games (Pruvost et al 2015) and associated with a graphic tablet to sonify handwriting as a remediation device for dysgraphic children (Danna et al 2015).…”