Relative timing of sound and vision: evaluation and correction

Speaker detection using the timing structure of lip motion and sound

“…We newly captured speech scenes of two persons, X and Y (the same persons in the training data), and segmented them into interval sequences 1 According to the research on the over all timing tolerance between video and audio by A. Peregudov et al [13], the thresholds of acceptability are about +90 ms (sound leading) to −185 ms (sound delayed). For these reason, we decided the standard deviation of the Gaussian, σ, as 2σ 100 ms ( the frame rate was 60 fps, therefore σ = 3 frames 50 ms ).…”

“…However, we human consider that these temporal differences are perfectly normal. In fact, it is known that some temporal variance is allowed in our speech perception [13]. Frame-wise integration methods are often used in speech recognition, however, they sometimes fail to describe such loose synchronization.…”