Database of speech and facial expressions recorded with optimized face motion capture settings

Abstract

The broad objective of the present research is the analysis of spoken English employing a multiplicity of modalities. An important stage of this process, discussed in the paper, is creating a database of speech accompanied with facial expressions. Recordings of speakers were made using an advanced system for capturing facial muscle motion. A brief historical outline, current applications, limitations and the ways of capturing face muscle motion as well as the problems with recording facial expressions are discussed. In particular, the scope of the present analysis concerns the registration of facial expressions related to emotions of speakers which accompany articulation. The camera system, instrumentation and software used for registration and for post-production are outlined. An analysis of the registration procedure and the results of the registration process was performed. The obtained results demonstrate how muscle movements can be registered employing reflective markers and point at the advantages and limitations of applying FMC (Face Motion Capture) technology in compiling a multimodal speech database. A short discussion pertaining to the usage of FMC as ground truth data source in facial expression databases concludes the paper.

Review on Mandibular Muscle Kinematics

“…Subsequently, from 2003 to 2006, they have not been noticed in this respect. The countries where the research has been developed, apart from those already mentioned, are: France [ 44 ]; Taiwan [ 45 , 46 , 47 ]; New Zealand [ 48 ]; Brazil [ 49 ]; Australia [ 50 ]; Poland [ 51 , 52 ]; Japan [ 53 , 54 ]; and Spain [ 55 ].…”

“…Most of the selected articles’ subjects were under-aged [ 39 , 42 , 43 , 45 , 46 , 47 , 50 , 53 , 56 , 57 , 60 ], and six publications included adults [ 39 , 40 , 48 , 51 , 52 , 54 , 59 ]. Only two research studies handled wide age ranges, basing their studies on children and adults [ 44 , 58 ], while four studies did not specify the sample [ 38 , 49 , 52 , 55 ]. The lowest sample size is one [ 48 , 55 ] and the highest 39 [ 58 ].…”

“…The minimum age is three years old and the maximum is 70 years old ( Figure 2 ). Of the total sample from all the studies registered, 293 were healthy individuals [ 38 , 39 , 40 , 41 , 42 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 52 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ] whereas there were only 58 subjects with speech disorders [ 43 , 56 , 57 , 60 ], 38 subjects were patients with cerebral palsy [ 45 , 46 , 47 , 50 ], one had previously been submitted to be studied for facial surgery with a muscle transplant [ 38 ], and 30 presented facial bone reconstruction with fibula graft or iliac crest [ 51 ].…”

Some consideration on expressive audiovisual speech corpus acquisition using a multimodal platform

“…Some information on corpus recording setups and statistics on their content can be found in the literature, but very little information can be found about some essential details for building an audiovi-Expressive audiovisual speech corpus acquisition 3 sual speech corpora efficiently. We have noticed that the majority of the existing descriptions are related to audiovisual corpora made for audiovisual speech recognition (a list of several corpora can be found in (Fernandez-Lopez and Sukno, 2018;Czyzewski et al, 2017)), very few describing corpora for audiovisual speech synthesis (Barra Chicote et al, 2008;Schabus and Pucher, 2014;Ouni et al, 2013;Mattheyses et al, 2009) and less using 3D tracking technologies (Busso et al, 2008;Kawaler and Czyzewski, 2019). As the main goal is creating a system for expressive audiovisual speech synthesis, the corpus should contain well recognizable emotions and should capture acoustic and visual speech articulation accurately.…”

“…To acquire facial data related to speech, optical motion-capture technique can be used (Ma et al, 2006;Vatikiotis-Bateson et al, 1993;Jiang et al, 2002;Kawaler and Czyzewski, 2019) for its good accuracy and appropriateness for kinematic analyses of lip/jaw movements (Feng and Max, 2014;.…”