WordSeg: Standardizing unsupervised word form segmentation from text

Abstract

A basic task in first language acquisition likely involves discovering the boundaries between words or morphemes in input where these basic units are not overtly segmented. A number of unsupervised learning algorithms have been proposed in the last 20 years for these purposes, some of which have been implemented computationally, but whose results remain difficult to compare across papers. We created a tool that is open source, enables reproducible results, and encourages cumulative science in this domain. WordSeg has a modular architecture: It combines a set of corpora description routines, multiple algorithms varying in complexity and cognitive assumptions (including several that were not publicly available, or insufficiently documented), and a rich evaluation package. In the paper, we illustrate the use of this package by analyzing a corpus of child-directed speech in various ways, which further allows us to make recommendations for experimental design of follow-up work. Supplementary materials allow readers to reproduce every result in this paper, and detailed online instructions further enable them to go beyond what we have done. Moreover, the system can be installed within container software that ensures a stable and reliable environment. Finally, by virtue of its modular architecture and transparency, WordSeg can work as an open-source platform, to which other researchers can add their own segmentation algorithms.

Segmentability Differences Between Child-Directed and Adult-Directed Speech: A Systematic Test With an Ecologically Valid Corpus

“…First, we used the automatic utterance boundaries provided by the LENA software (“A,” short for “automatic boundaries”), as well as combined together the text from segments labeled as continuations of each other by coders (“H” for “human boundaries”). Second, since performance is dependent on corpus size (see Bernard et al, 2018), we had three versions of each CDS corpus: the full one, a shortened CDS corpus to match the ADS corpus in number of words, and a shortened CDS corpus to match the ADS corpus in number of utterances. After crossing these two factors, performance could be compared between, on the one hand, ADS-A/H (ADS with automatic or human utterance boundaries), and, on the other hand, one of (1) CDS-A/H-full (corresponding full CDS corpus), (2) CDS-A/H-WM (cut at the same number of word tokens found in the corresponding ADS), or (3) CDS-A/H-UM (cut at the same number of utterances).…”

“…Each algorithm (with default parameters, except as noted below) was run using the WordSeg package (Bernard et al, 2018), which also performs the evaluation. Due to space restrictions, we cannot provide fuller descriptions here, but we refer readers to Bernard et al (2018), where the algorithms and the evaluation are explained. In a nutshell, both training and evaluation are done over the whole corpus because these algorithms are unsupervised, and thus there is no risk of overfitting.…”

Is Word Segmentation Child’s Play in All Languages?

“…For lack of space, we will only briefly describe the algorithms drawn from WordSeg (see Johnson and Goldwater 2009;Monaghan and Christiansen 2010;Lignos 2012;Daland and Zuraw 2013;Saksida et al 2017;Bernard et al 2018). All algorithms were used with their default parameters.…”

“…Most previous computational research has used as input texts representing phonologized language, that is, sequences of phonemes with no overt word boundaries, and the task is to retrieve these. Several algorithms inspired by laboratory research on infant word segmentation are currently represented in WordSeg, an open source package (Bernard et al, 2018). Are such algorithms as robust to cross-linguistic variation as human infants are?…”

“…Because of time constraints, only the first 50000 utterances of the three largest corpora, Turkish, Russian and Indonesian, were segmented by AG. This would play a negligible role in results, since variation in corpus size beyond the first 5k utterances does not affect performance of this segmentation system(Bernard et al, 2018).…”