The analysis of lectures and meetings inside smart rooms has recently attracted much interest in the literature, being the focus of international projects and technology evaluations. A key enabler for progress in this area is the availability of Ambrish Tyagi has contributed to this work during two summer internships with the IBM T.appropriate multimodal and multi-sensory corpora, annotated with rich human activity information during lectures and meetings. This paper is devoted to exactly such a corpus, developed in the framework of the European project CHIL, ''Computers in the Human Interaction Loop''. The resulting data set has the potential to drastically advance the state-of-the-art, by providing numerous synchronized audio and video streams of real lectures and meetings, captured in multiple recording sites over the past 4 years. It particularly overcomes typical shortcomings of other existing databases that may contain limited sensory or monomodal data, exhibit constrained human behavior and interaction patterns, or lack data variability. The CHIL corpus is accompanied by rich manual annotations of both its audio and visual modalities. These provide a detailed multi-channel verbatim orthographic transcription that includes speaker turns and identities, acoustic condition information, and named entities, as well as video labels in multiple camera views that provide multi-person 3D head and 2D facial feature location information. Over the past 3 years, the corpus has been crucial to the evaluation of a multitude of audiovisual perception technologies for human activity analysis in lecture and meeting scenarios, demonstrating its utility during internal J. Turmo

We propose a system for detecting the active speaker in cluttered and reverberant environments where more than one person speaks and moves. Rather than using only audio information, the system utilizes audiovisual information from multiple acoustic and video sensors that feed separate audio and video tracking modules. The audio module operates using a particle filter (PF) and an information-theoretic framework to provide accurate acoustic source location under reverberant conditions. The video subsystem combines in 3-D a number of 2-D trackers based on a variation of Stauffer's adaptive background algorithm with spatiotemporal adaptation of the learning parameters and a Kalman tracker in a feedback configuration. Extensive experiments show that gains are to be expected when fusion of the separate modalities is performed to detect the active speaker.

Abstract-Estimating the direction of arrival (DOA) of an acoustic source relies on the successful estimation of the relative delay between pairs of microphone signals. Processing is performed at the current time by operating on blocks of recorded data. When these recordings are performed in environments of strong multipath reflections, algorithms often fail to distinguish between the true DOA and that of a dominant reflection. In this letter, we assume Gaussianity of the source signal and use an information-theoretical measure, often met in blind source separation algorithms, to derive a robust DOA estimator, even under significant reverberant conditions. We discuss the most popular algorithm for time delay estimation, namely, the generalized cross-correlation method, and demonstrate under certain conditions its connection to the proposed one. Performance is demonstrated for both algorithms with sets of simulated results as a function of different reverberation times, microphone spacing, and data block size. The results indicate that the examined framework can accurately track the DOA of a typical acoustic source.

A theoretical framework is established, for the robustness of multichannel sound equalization in reverberant environments. Using results from statistical room acoustics, a closed-form expression is derived that predicts the degradation in performance of an equalization system as the sound source moves from its nominal position inside the enclosure. The presented analysis also provides means of identifying the performance bounds that can be expected when using such a system in an actual room. Using extensive computer simulations, the effect of physical parameters such as the relative positions of the source and the receivers, as well as effects of different design parameters are investigated. Based on the conditions imposed by these parameters, it is shown that, depending on the array geometry and the exact form of the equalizers, slight performance gains can be expected as the number of receivers is increased.

No abstract