Lip Assistant: Visualize Speech for Hearing Impaired People in Multimedia Services

Abstract

This paper presents a very low bit rate speechto-video synthesizer, named lip assistant, to help hearing impaired people to better access multimedia services via lipreading. Lip assistant can automatically convert acoustic speech to lip parameters with a bit rate of 2.2kbps, and decode them to video-realistic mouth animation on the fly. We use multi-stream HMMs (MSHMMs) and the principal component analysis (PCA) to model the audio-visual speech and the visual articulations, which are learned from AV facial recordings. Speech is converted to lip parameters with natural dynamics by an expectation maximization (EM)-based audioto-lip converter. The video synthesizer generates video-realistic mouth animations from the encoded lip parameters via PCA expansion. Finally, mouth animation is superimposed on the original video as an assistant for hearing impaired viewers to make a better understanding on the audio-visual contents. Experimental results shows that lip assistant can significantly improve the speech intelligibility of both machines and humans.

A Framework for Speechreading Acquisition Tools

“…Both examples of Spectrograms are analytic aids, and provide high information due to the richness of the visualised data. Lip Assistant [54] is an SAT that generates magnified realistic animations of a speaker's lips that are superimposed on the bottom left of a video. Lip Assistant is analytic as it focusses exclusively on lip shapes and it provides low information.…”