Toward Perceptually Realistic Talking Heads: Models, Methods, and McGurk

Cosker, Darren; Paddock, Susan M.; Marshall, David; Rosin, Paul L.; Rushton, Simon K.

doi:10.1145/1077399.1077405

Cited by 19 publications

(15 citation statements)

References 18 publications

(30 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…An interesting approach to this issue was taken in the work by [Cosker et al 2004], in which the quality of their two-dimensional facial animation system was tested using a well-known perceptual conflict (the 'McGurk-effect') between the auditory and visual senses. Their evaluation thus relied on an indirect, perceptual measure rather than on an explicit judgment of realism.…”

Section: Facial Animation and Realismmentioning

confidence: 99%

Psychophysical evaluation of animated facial expressions

Wallraven

Breidt

Cunningham

et al. 2005

Proceedings of the 2nd Symposium on Applied Perception in Graphics and Visualization

View full text Add to dashboard Cite

The human face is capable of producing an astonishing variety of expressions-expressions for which sometimes the smallest difference changes the perceived meaning noticably. Producing realisticlooking facial animations that are able to transport this degree of complexity continues to be a challenging research topic in computer graphics. One important question that remains to be answered is: When are facial animations good enough? Here we present an integrated framework in which psychophysical experiments are used in a first step to systematically evaluate the perceptual quality of computer-generated animations with respect to real-world video sequences. The result of the first experiment is an evaluation of several animation techniques in which we expose specific animation parameters that are important for perceptual fidelity. In a second experiment we then use these benchmarked animations in the context of perceptual research in order to systematically investigate the spatio-temporal characteristics of expressions. Using such an integrated approach, we are able to provide insights into facial expressions for both the perceptual and computer graphics community.

show abstract

Section: Facial Animation and Realismmentioning

confidence: 99%

Psychophysical evaluation of animated facial expressions

Wallraven

Breidt

Cunningham

et al. 2005

Proceedings of the 2nd Symposium on Applied Perception in Graphics and Visualization

View full text Add to dashboard Cite

show abstract

“…An established body of research already focuses on quantifying the intelligibility-benefit or realism of animated talking heads, ideally as compared to a video-realistic standard (Ouni et al, 2007;Cosker et al, 2005). However, as the articulators that researchers/teachers wish to present to their subjects in the aforementioned scenario are generally outside the line of sight, these evaluation methods cannot be directly applied to intra-oral visualizations.…”

Section: Introductionmentioning

confidence: 99%

Automatic Detection of Intra-Word Code-Switching

Nguyen¹,

Cornips²

2016

Proceedings of the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology

View full text Add to dashboard Cite

Welcome to the 14th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology. The workshop aims to bring together researchers interested in applying computational techniques to problems in morphology, phonology, and phonetics. Our program this year highlights the ongoing and important interaction between work in computational linguistics and work in theoretical linguistics. We received 23 submissions and accepted 11.The volume of submissions made it necessary to recruit several additional reviewers. We'd like to thank all of these people for agreeing to review papers on what seemed like impossibly short notice.This year also marks the first SIGMORPHON shared task, on morphological reinflection. The shared task received 9 submissions, all of which were accepted, and greatly advanced the state of the art in this area.We thank all the authors, reviewers and organizers for their efforts on behalf of the community. AbstractThis paper conceptualizes speech prosody data mining and its potential application in data-driven phonology/phonetics research. We first conceptualize Speech Prosody Mining (SPM) in a time-series data mining framework. Specifically, we propose using efficient symbolic representations for speech prosody time-series similarity computation. We experiment with both symbolic and numeric representations and distance measures in a series of time-series classification and clustering experiments on a dataset of Mandarin tones. Evaluation results show that symbolic representation performs comparably with other representations at a reduced cost, which enables us to efficiently mine large speech prosody corpora while opening up to possibilities of using a wide range of algorithms that require discrete valued data. We discuss the potential of SPM using time-series mining techniques in future works. IntroductionCurrent investigations on the phonology of intonation and tones (or pitch accent) typically employ data-driven approaches by building research on top of manual annotations of a large amount of speech prosody data (for example, (Morén and Zsiga, 2006; Zsiga and Zec, 2013), and many others). Meanwhile, researchers are also limited by the amount of resources invested in such expensive endeavor of manual annotations. Given this paradox, we believe that this type of data driven approach in phonology-phonetics interface can benefit from tools that can efficiently index, query, classify, cluster, summarize, and discover meaningful prosodic patterns from a large speech prosody corpus.The data mining of f 0 1 (pitch) contour patterns from audio data has recently gained success in the domain of Music Information Retrieval (aka MIR, see (Gulati and Serra, 2014; Gulati et al., 2015; Ganguli, 2015) for examples). In contrast, the data mining of speech prosody f 0 data (here on referred to as Speech Prosody Mining (SPM) 2 ) is a less explored research topic (Raskinis and Kazlauskiene, 2013). Fundamentally, SPM in a large prosody corpus aims at discovering meaningful patterns in the f ...

show abstract

“…Other methods that attempt to quantity the quality of animation are limited to learning based methods [Ma and Deng 2012] or comparisons between the original data and the reconstructed data, which are not applicable to animator-driven methods. McGurk studies [Cosker et al 2005] can be performed to see how the McGurk effect on real humans compares to that of synthesized speech. Likewise, noise studies where words are randomly removed from utterances and must be recovered by reading lips can help judge the quality of the articulation.…”

Section: Direct Methods Comparisonsmentioning

confidence: 99%

A Practical and Configurable Lip Sync Method for Games

Feng

Marsella

et al. 2013

Proceedings of Motion on Games

View full text Add to dashboard Cite

We demonstrate a lip animation (lip sync) algorithm for real-time applications that can be used to generate synchronized facial movements with audio generated from natural speech or a text-to-speech engine. Our method requires an animator to construct animations using a canonical set of visemes for all pairwise combinations of a reduced phoneme set (phone bigrams). These animations are then stitched together to construct the final animation, adding velocity and lip-pose constraints. This method can be applied to any character that uses the same, small set of visemes. Our method can operate efficiently in multiple languages by reusing phone bigram animations that are shared among languages, and specific word sounds can be identified and changed on a per-character basis. Our method uses no machine learning, which offers two advantages over techniques that do: 1) data can be generated for non-human characters whose faces can not be easily retargeted from a human speaker's face, and 2) the specific facial poses or shapes used for animation can be specified during the setup and rigging stage, and before the lip animation stage, thus making it suitable for game pipelines or circumstances where the speech targets poses are predetermined, such as after acquisition from an online 3D marketplace. Synchronizing the lip and mouth movements naturally along with animation is an important part of convincing 3D character performance. In this paper, we present a simple, portable and editable lip-synchronization method that works for multiple languages, requires no machine learning, can be constructed by a skilled animator, is effective for real-time simulations such as games, and can be personalized for each character. Our method associates animation curves designed by an animator on a fixed set of static facial poses, with sequential pairs of phonemes (phone bigrams), and then stitches these animations together to create a set of curves for the facial poses along with constraints that ensure that key poses are properly played. Diphone-and triphone-based methods have been explored in various previous works, often requiring machine learning. However, our experiments have shown that animating phoneme pairs (such as phone bigrams or diphones), as opposed to phoneme triples or longer sequences of phonemes, is sufficient for many types of animated characters. Also, our experiments have shown that skilled animators can sufficiently generate the data needed for good quality results. Thus our algorithm does not need any specific rules about coarticulation, such as dominance functions or language rules. Such rules are implicit within the artist-produced data. In order to produce a tractible set of data, our method reduces

show abstract

Toward Perceptually Realistic Talking Heads: Models, Methods, and McGurk

Cited by 19 publications

References 18 publications

Psychophysical evaluation of animated facial expressions

Psychophysical evaluation of animated facial expressions

Automatic Detection of Intra-Word Code-Switching

A Practical and Configurable Lip Sync Method for Games

Contact Info

Product

Resources

About