No abstract

Active speaker detection (ASD) seeks to detect who is speaking in a visual scene of one or more speakers. The successful ASD depends on accurate interpretation of short-term and long-term audio and visual information, as well as audio-visual interaction. Unlike the prior work where systems make decision instantaneously using short-term features, we propose a novel framework, named TalkNet, that makes decision by taking both short-term and long-term features into consideration. TalkNet consists of audio and visual temporal encoders for feature representation, audio-visual cross-attention mechanism for inter-modality interaction, and a self-attention mechanism to capture long-term speaking evidence. The experiments demonstrate that TalkNet achieves 3.5% and 2.2% improvement over the state-of-the-art systems on the AVA-ActiveSpeaker dataset and Columbia ASD dataset, respectively. Code has been made available at: https://github.com/TaoRuijie/TalkNet_ASD. CCS CONCEPTS• Information systems → Speech / audio search.

We propose an audiovisual fusion algorithm for 3D speaker tracking from a localised multi-modal sensor platform composed of a camera and a small microphone array. After extracting audiovisual cues from individual modalities we fuse them adaptively using their reliability in a particle filter framework. The reliability of the audio signal is measured based on the maximum Global Coherence Field (GCF) peak value at each frame. The visual reliability is based on colour-histogram matching with detection results compared with a reference image in the RGB space. Experiments on the AV16.3 dataset show that the proposed adaptive audiovisual tracker outperforms both the individual modalities and a classical approach with fixed parameters in terms of tracking accuracy.

Most of the prior studies in the spatial Direction of Arrival (DoA) domain focus on a single modality. However, humans use auditory and visual senses to detect the presence of sound sources. With this motivation, we propose to use neural networks with audio and visual signals for multi-speaker localization. The use of heterogeneous sensors can provide complementary information to overcome uni-modal challenges, such as noise, reverberation, illumination variations, and occlusions. We attempt to address these issues by introducing an adaptive weighting mechanism for audio-visual fusion. We also propose a novel video simulation method that generates visual features from noisy target 3D annotations that are synchronized with acoustic features. Experimental results confirm that audio-visual fusion consistently improves the performance of speaker DoA estimation, while the adaptive weighting mechanism shows clear benefits.

In this paper we present a new technique for determining the angle of incident light using an image sensor. This technique makes use of two distinct pixel types for simultaneous coarse angle estimation and fine angle detection. A new type of pixel structure, the quadrature pixel cluster (QPC) is introduced, which produces response that varies linearly with incident light angle. The proposed technique greatly reduces the overall sensor complexity and is very area efficient compared to the previous work on angle detection. The sensor is fabricated in 65 nm mixedsignal CMOS process and can accurately distinguish between angles in the range from -35 • to +35 • . Index Terms-Angle detection; linear angle sensitive pixels; quadrature pixel cluster; light field sensor; track-and-tune angle detection; CMOS image sensor.This 5D function can be further reduced to a 4D one by considering the fact that the radiance along any ray does not change unless blocked [3][4]. This 4D representation can be used to completely describe any visual scene around us. Conventional image sensors sample only a 2D version of the imaging scene and hence are limited in their capabilities for post capture image processing. Extraction of the complete 4D parameters from light field allows for certain image processing Vigil Varghese, Xinyuan Qian and Shoushun Chen are with VIRTUS