Look who’s talking: A comparison of automated and human-generated speaker tags in naturalistic day-long recordings

Abstract

The LENA system has revolutionized research on language acquisition, providing both a wearable device to collect daylong recordings of children's environments, and a set of automated outputs that process, identify, and classify speech using proprietary algorithms. This output includes information about input sources (e.g., adult male, electronics). While this system has been tested across a variety of settings, here we delve deeper into validating the accuracy and reliability of LENA's automated diarization, i.e., tags of who is talking. Specifically, we compare LENA's output with a gold standard set of manually generated talker tags from a dataset of 88 day-long recordings, taken from 44 infants at 6 and 7 months, which includes 57,983 utterances. We compare accuracy across a range of classifications from the original Lena Technical Report, alongside a set of analyses examining classification accuracy by utterance type (e.g., declarative, singing). Consistent with previous validations, we find overall high agreement between the human and LENA-generated speaker tags for adult speech in particular, with poorer performance identifying child, overlap, noise, and electronic speech (accuracy range across all measures: 0-92%). We discuss several clear benefits of using this automated system alongside potential caveats based on the error patterns we observe, concluding with implications for research using LENA-generated speaker tags.

A thorough evaluation of the Language Environment Analysis (LENA) system

“…As we summarize below, this approach has many advantages, which may explain its expanding popularity. While over a hundred papers over the past two decades have used the output automatically provided by LENA , only a handful include validity estimates (e.g., d 'Apice et al 2019;Weisleder & Fernald 2013;Zimmerman et al 2009), even fewer where validity estimation was the primary focus of the paper (e.g., Bulgarelli & Bergelson 2019;Busch et al 2018;Canault et al 2016;Ganek & Eriks-Brophy 2018;Lehet et al 2018). As a result, few studies report sufficient details about validation accuracy for one or more metrics, limiting the interpretability of the results of a meta-analytic assessment (cf.…”

Analysis of children’s everyday language experiences using longform audio: promises and pitfalls (Análisis de las experiencias lingüísticas cotidianas de niños y niñas utilizando audio de formato largo: posibles ventajas y dificultades)

“…However, other factors may systematically affect accuracy. For example, read speech may show greater accuracy than singing (Bulgarelli & Bergelson, 2019), and there is emerging evidence of variation in accuracy across time of day (Lehet et al, 2020). There is also ample evidence for complex gender effects.…”

Using a developmental-ecological approach to understand the relation between language and music

“…The CHILDES database (MacWhinney & Snow, 1990) is a corpus of transcribed and tagged speech to children in both lab and naturalistic settings that has been used to create models of child language learning (e.g., Mintz, 2003). More recently, there has been a renewed interest in collecting day-long, longitudinal audio/visual recordings to describe the nuances of language input in first years of life (e.g., Bulgarelli & Bergelson, 2020,Casillas et al, 2020, Roy et al 2015, Sullivan et al, 2021,Tamis-LeMonda et al, 2019. This work has led to novel findings, such as the role of context (Roy et al, 2015) and temporal structure (Casillas et al, 2020) in language development.…”