A comparative study of English viseme recognition methods and algorithms

Abstract

An elementary visual unit -the viseme is concerned in the paper in the context of preparing the feature vector as a main visual input component of Audio-Visual Speech Recognition systems. The aim of the presented research is a review of various approaches to the problem, the implementation of algorithms proposed in the literature and a comparative research on their effectiveness. In the course of the study an optimal feature vector construction and an appropriate selection of the classifier were sought. The experimental research was conducted on the basis of a spoken corpus in which speech was represented both acoustically and visually. The extracted features represented three types: geometrical, textural and mixed ones. The features were processed employing the classification algorithms based on Hidden Markov Models and Sequential Minimal Optimization. Tests were carried out employing the processed video material recorded with English native speakers who read specially prepared list of commands. The obtained results are discussed in the paper.

Harnessing AI for Speech Reconstruction using Multi-view Silent Video Feed

“…That is because characters like 'p' and 'b' belong to the same viseme class. Similar is the case with expressions like "Elephant Juice" and "I love you" which though having similar visemic appearances definitely have very different sounds and meanings [26].…”

The timecourse of multisensory speech processing in unilaterally stimulated cochlear implant users revealed by ERPs

“…A viseme is the visual equivalent of the phoneme: a static image of a person articulating a phoneme ( Dong et al, 2003 ). There are some phonemes that share identical visemes ( Cappelletta and Harte, 2012 , Lucey et al, 2004 , Mahavidyalaya, 2014 ), but for the vowels of the syllables used in this study, the visemes are clearly distinguishable (see illustrations in Jachimski et al, 2018 ), which is of importance given that we present visual-only trials as well. The syllables were edited using Audacity (version 3.0.2) in order to be cut and adjusted to the same duration of 400 ms.…”

An Attention Self-Supervised Contrastive Learning Based Three-Stage Model for Hand Shape Feature Representation in Cued Speech

“…The specific data is shown in Table 1. The new British English dataset contains 11 visemes [27], 8 hand shapes and 4 hand positions to encode 17 vowels and 24 consonants. RGB video images of the interpreter's upper body are available at 25 fps, and the spatial resolution is 720 × 1280.…”

“…Ours(multi) represents our model in the multi-speaker scenario. +S3 23 27. 30.48 27.36 26.20 46.79 46.60 Ours-SANs 21.8 27.21 25.41 25.96 43.89 45.42 Ours 20.37 26.16 23.39 24.88 38.42 42.02 Ours(multi) -19.93 17.64 14.59 20.77 26.38 5.…”