A corpus of Schlieren photography of speech production: potential methodology to study aerodynamics of labial, nasal and vocalic processes

Tomaschek, Fabian; Arnold, Denis; Sering, Konstantin; Strauss, Friedolin

doi:10.1007/s10579-021-09550-8

Cited by 1 publication

(2 citation statements)

References 20 publications

(21 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, when synergized with machine learning, this method can devise optimal, patient-specific therapy to correct speech disorders. It is noted that while several studies have used schlieren imaging to study speech flows [ 26 , 29 , 30 ], none have applied it to speech therapy, focusing instead on the flow intensity of nasal/oral vowels in different languages (German [ 26 ], Polish [ 29 ], and French [ 30 ]). Harvey et al [ 31 ] attempted to recover physical audio signals from high-speed schlieren images, but this approach may be limited to shockwaves generated by events such as clapping hands, snapping belts, and cracking towels, where high-frequency signals dominate.…”

Section: Discussionmentioning

confidence: 99%

“…Schlieren imaging has been used in a limited number of studies focused on speech production. Tomaschek et al [ 26 ] recorded the flow dynamics from lip, nasal, and vocal speeches using a schlieren system and quantitatively compared the flow intensity nasal and non-nasal sounds, revealing that delayed uvular closure nasalized the vowel sound. Furthermore, the flow variations following lip closure, as described in other studies [ 27 , 28 ].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Schlieren imaging and video classification of alphabet pronunciations: exploiting phonetic flows for speech recognition and speech therapy

Talaat,

Barari,

et al. 2024

Vis. Comput. Ind. Biomed. Art

View full text Add to dashboard Cite

Speech is a highly coordinated process that requires precise control over vocal tract morphology/motion to produce intelligible sounds while simultaneously generating unique exhaled flow patterns. The schlieren imaging technique visualizes airflows with subtle density variations. It is hypothesized that speech flows captured by schlieren, when analyzed using a hybrid of convolutional neural network (CNN) and long short-term memory (LSTM) network, can recognize alphabet pronunciations, thus facilitating automatic speech recognition and speech disorder therapy. This study evaluates the feasibility of using a CNN-based video classification network to differentiate speech flows corresponding to the first four alphabets: /A/, /B/, /C/, and /D/. A schlieren optical system was developed, and the speech flows of alphabet pronunciations were recorded for two participants at an acquisition rate of 60 frames per second. A total of 640 video clips, each lasting 1 s, were utilized to train and test a hybrid CNN-LSTM network. Acoustic analyses of the recorded sounds were conducted to understand the phonetic differences among the four alphabets. The hybrid CNN-LSTM network was trained separately on four datasets of varying sizes (i.e., 20, 30, 40, 50 videos per alphabet), all achieving over 95% accuracy in classifying videos of the same participant. However, the network’s performance declined when tested on speech flows from a different participant, with accuracy dropping to around 44%, indicating significant inter-participant variability in alphabet pronunciation. Retraining the network with videos from both participants improved accuracy to 93% on the second participant. Analysis of misclassified videos indicated that factors such as low video quality and disproportional head size affected accuracy. These results highlight the potential of CNN-assisted speech recognition and speech therapy using articulation flows, although challenges remain in expanding the alphabet set and participant cohort.

show abstract