A functional head-movement corpus and convolutional neural networks (CNNs) for detecting head-movement functions are presented for analyzing the multiple communicative functions of head movements in multiparty face-to-face conversations. First, focusing on the multifunctionality of head movements, i.e., that a single head movement can simultaneously perform multiple functions, this paper defines 32 nonmutually-exclusive function categories, whose genres are speech production, eliciting and giving feedback, turn management, and cognitive and affect display. To represent and capture arbitrary multifunctional structures, our corpus employs multiple binary codes and logical-sum-based aggregations of multiple coders' judgments. A corpus analysis targeting four-party Japanese conversations revealed multifunctional patterns in which the speaker modulates multiple functions, such as emphasis and eliciting listeners' responses, through rhythmic head movements, and listeners express various attitudes and responses through continuous back-channel head movements. This paper proposes CNN-based binary classifiers for detecting each of the functions from the angular velocity of the head pose and the presence or absence of utterances. The experimental results showed that the recognition performance varies greatly, from approximately 30% to 90% in terms of the F-score, depending on the function category, and the performance was positively correlated with the amount of data and inter-coder agreement. In addition, we noted a tendency toward overdetection that added more functions to those originally in the corpus. The analyses and experiments confirm that our approach is promising for studying the multifunctionality of head movements. INDEX TERMS deep neural networks, gesture recognition, meeting analysis, multimodal sensor, nonverbal behaviors, social signal processing This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.
The compressed ultrafast photography (CUP) method is used to observe ultrafast light emission phenomena by restoring multiple images from a single observed image via a compressed sensing algorithm. However, because its regularization function is only suitable for ultrafast light emissions with lattice contours, the CUP method frequently produces artifacts in the restoration result. To solve this problem, we propose a restoration method that is suitable for ultrafast light emissions with any contour shapes. Specifically, we derive a regularization function that automatically estimates the contours of the ultrafast light emissions. Furthermore, we correct the movement of the ultrafast light emissions. By solving the inverse problem with the derived regularization function, accurate restoration results without artifacts can be obtained. Simulations using datasets that emulate fundamental phenomena show that the proposed method is superior to the conventional CUP method in terms of visual quality and the correlation with the original image.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.