Deep multiple instance learning for foreground speech localization in ambient audio from wearable devices

Abstract

Over the recent years, machine learning techniques have been employed to produce state-of-the-art results in several audio related tasks. The success of these approaches has been largely due to access to large amounts of open-source datasets and enhancement of computational resources. However, a shortcoming of these methods is that they often fail to generalize well to tasks from real life scenarios, due to domain mismatch. One such task is foreground speech detection from wearable audio devices. Several interfering factors such as dynamically varying environmental conditions, including background speakers, TV, or radio audio, render foreground speech detection to be a challenging task. Moreover, obtaining precise moment-to-moment annotations of audio streams for analysis and model training is also time-consuming and costly. In this work, we use multiple instance learning (MIL) to facilitate development of such models using annotations available at a lower time-resolution (coarsely labeled). We show how MIL can be applied to localize foreground speech in coarsely labeled audio and show both bag-level and instance-level results. We also study different pooling methods and how they can be adapted to densely distributed events as observed in our application. Finally, we show improvements using speech activity detection embeddings as features for foreground detection.

Does Your Smartphone “Know” Your Social Life? A Methodological Comparison of Day Reconstruction, Experience Sampling, and Mobile Sensing

“…This might in part be due to technical limitations of the used MS algorithm: Although the algorithm achieved high accuracies of more than 85% in prior studies (Lane et al, 2012; Rabbi et al, 2011), the algorithm’s accuracy in less controlled environments is probably lower, as indicated by the size of agreement with DRM and ESM in the current study. In the future, researchers will likely have access to more sophisticated algorithms—for example, first evidence suggests that algorithms based on a distinction of foreground versus background sound might outperform more traditional voice-detection algorithms (Hebbar et al, 2021).…”

“…In general, we expected the agreement between DRM and MS to be lower than the agreement between ESM and MS because of the greater time delay and increased memory biases of DRM compared with ESM and MS. We further expected DRM and ESM to agree more on face-to-face interactions than DRM and MS or ESM and MS because of a closer alignment of operationalizations (e.g., social interactions assessed in DRM and MS may include periods without conversation) and because of technical challenges of MS, such as accurately identifying speakers (e.g., the participant or a surrounding group of people) and filtering out background noise (Hebbar et al, 2021). Accordingly, we derived the following hypotheses:…”

Mobile Sensing Methods

“…It is not uncommon for an EAR study to accrue hundreds of hours of audio data. These audio data then, at least for the moment, need to be listened to and behaviorally coded by human coders (see Dubey et al, 2016;Hebbar et al, 2021;Schindler et al, 2022, for recent proof-of-concept attempts to automate aspects of the coding). In that, then, the EAR, as a naturalistic observation method, is ultimately subject to at least some of the same challenges that lab-based observation is.…”

A theoretical analysis based on causal inference and single-instance learning

“…Multi-instance learning (MIL) was originally used for the field of hand-printed numerals identification [ 1 ] and drug activity prediction [ 2 ]. Instead of considering a series of individually labeled instances, MIL focuses on the labels of sets (or called bags ) of instances and demonstrate strong capabilities in many areas [ 3 ], e.g., speech localization [ 4 ], entity classification [ 5 ], protein structure determination [ 6 ], biometric authentication system [ 7 – 10 ], human pose estimation [ 11 ], medical image analysis [ 12 ], understanding chest CT imaging of COVID-19 [ 13 ], and clinical outcome prediction of COVID-19 [ 14 ].…”