Assessing face and speech consistency for monologue detection in video

Nock, Harriet J.; Iyengar, G.; Neti, C.

doi:10.1145/641043.641070

Cited by 9 publications

(20 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Synchronization has been studied in both audio and video signals [6,1,19,17,5,7]. For instance, synchrony measures have been derived [17,7] for associating the movement of mouths to the oscillation of sound waves.…”

Section: Literature Reviewmentioning

confidence: 99%

Detecting motion synchrony by video tubes

Zheng

Tomasi

2011

Proceedings of the 19th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Motion synchrony, i.e., the coordinated motion of a group of individuals, is an interesting phenomenon in nature or daily life. Fish swim in schools, birds fly in flocks, soldiers march in platoons, etc. Our goal is to detect motion synchrony that may be present in the video data, and to track the group of moving objects as a whole. This opens the door to novel algorithms and applications. To this end, we model individual motions as video tubes in space-time, define motion synchrony by the geometric relation among video tubes, and track a whole set of tubes by dynamic programming. The resulting algorithm is highly efficient in practice. Given a video clip of T frames of resolution X × Y , we show that finding the K spatially correlated video tubes and determining the presence of synchrony can be solved optimally in O(XY T K) time. Preliminary experiments show that our method is both effective and efficient. Typical running times are 30 − 100 VGA-resolution frames per second after feature extraction, and the accuracy for the detection of synchrony is more than 90% as evaluated in our annotated data set.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Detecting motion synchrony by video tubes

Zheng

Tomasi

2011

Proceedings of the 19th ACM International Conference on Multimedia

View full text Add to dashboard Cite

show abstract

“…They localized the speaker in the image by computing mutual information between the signals. This method has extended and has been applied to especially sound source localization problem [14] [15]. A limitation of the method is the assumption that the target does not move in the images.…”

Section: Sensor Integration Based On Evaluating Synchrony Between mentioning

confidence: 99%

People tracking by fusing different kinds of sensors, floor sensors and acceleration sensors

Ikeda

Ishiguro

Nishimura

2006

2006 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems

View full text Add to dashboard Cite

To realize accurate tracking of people in the environment, many studies have been proposed using vision sensors, floor sensors, and wearable devices. Floor sensors can reliably detect current positions, but it is difficult to estimate correct associations between observations and multiple people. To solve the problem, we propose to combine acceleration sensors that are attached to the human body. Since the signals from floor sensors and acceleration sensors synchronize when they observe same person walking, these signals are not independent. The synchrony between the signals is evaluated based on statistical test to find correct association. People tracking examples are shown to confirm the effectiveness of the proposed method. Significant improvement in correct association rate is achieved compared to the results only by floor sensors.

show abstract

“…It affects methods based on MI as well [13]. Hence, very small images O(50 × 50) have been commonly used [3,20,23,25], out of which only a few dozen features were selected by aggressive pruning or face detection steps (the latter limiting audio analysis to speech). In contrast, we seek localization of general unknown audiovisual sources, while handling intricate details and motion.…”

Section: Canonical Correlation: Limitationsmentioning

confidence: 99%

“…A particularly interesting sensor combination involves visual motion in conjunction with associated audio. Activity in computer vision involving audio analysis has various research aspects [4,26], including lip reading [3,25], analysis and synthesis of music from motion [22], audio filtering based on motion [6], and source separation based on vision [14,17,20,23,27]. We note that physiological evidence and analysis of biological systems show that fusion of audio-visual information is used to enhance perception [9,12,16].…”

Section: Introductionmentioning

confidence: 99%

“…These pixels should be distinguished from other moving objects. We do not limit the problem to talking faces [3,4,20,23] or other specific classes of sources [22], but seek a general and effective algorithm to achieve this goal. Some existing methods use several microphones (emulating binaural hearing), where stereo triangulation indicates the spatial location of the sources [2,24,28].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Pixels that Sound

Kidron

Schechner

Elad

2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05)

139

115

View full text Add to dashboard Cite

People and animals fuse auditory and visual information to obtain robust perception. A particular benefit of such cross-modal analysis is the ability to localize visual events associated with sound sources. We aim to achieve this using computer-vision aided by a single microphone. Past efforts encountered problems stemming from the huge gap between the dimensions involved and the available data. This has led to solutions suffering from low spatio-temporal resolutions. We present a rigorous analysis of the fundamental problems associated with this task. Then, we present a stable and robust algorithm which overcomes past deficiencies. It grasps dynamic audio-visual events with high spatial resolution, and derives a unique solution. The algorithm effectively detects pixels that are associated with the sound, while filtering out other dynamic pixels. It is based on canonical correlation analysis (CCA), where we remove inherent ill-posedness by exploiting the typical spatial sparsity of audio-visual events. The algorithm is simple and efficient thanks to its reliance on linear programming and is free of user-defined parameters. To quantitatively assess the performance, we devise a localization criterion. The algorithm capabilities were demonstrated in experiments, where it overcame substantial visual distractions and audio noise.

show abstract

Assessing face and speech consistency for monologue detection in video

Cited by 9 publications

References 0 publications

Detecting motion synchrony by video tubes

Detecting motion synchrony by video tubes

People tracking by fusing different kinds of sensors, floor sensors and acceleration sensors

Pixels that Sound

Contact Info

Product

Resources

About