Accuracy of Four Language Analysis Procedures Performed Automatically

Abstract

Most software for language analysis has relied on an interaction between the metalinguistic skills of a human coder and the calculating ability of the machine to produce reliable results. However, probabilistic parsing algorithms are now capable of highly accurate and completely automatic identification of grammatical word classes. The program Computerized Profiling combines a probabilistic parser with modules customized to produce four clinical grammatical analyses: MLU, LARSP, IPSyn, and DSS. The accuracy of these analyses was assessed on 69 language samples from typically developing, speech-impaired, and language-impaired children, 2 years 6 months to 7 years 10 months. Values obtained with human coding and by the software alone were compared. Results for all four analyses produced automatically were comparable to published data on the manual interrater reliability of these procedures. Clinical decisions based on cutoff scores and productivity data were little affected by the use of automatic rather than human-generated analyses. These findings bode well for future clinical and research use of automatic language analysis software.

Development of Linguistic Linked Open Data Resources for Collaborative Data-Intensive Research in the Language Sciences

“…It is well acknowledged that good LSA can be quite time-consuming (Overton and Wren 2014). Some studies have estimated that it takes up to 8 hours of training and from 45 minutes to one hour of work after a transcript has been generated to compute DSS (Long and Channell 2001;Cochran and Masterson 1995). One study (Gorman 2010) estimated that it takes more than 30 minutes per sample following transcription to compute the Index of Productive Syntax (IPSYN; Scarborough 1990).…”

Young Children's Structure Production: A Revision of the Index of Productive Syntax

“…yntactic measurement is an important tool in understanding language development. Since the 1970s, measures of syntax derived from natural language samples have proven useful, despite some shortcomings, for describing young children's syntactic proficiency and growth (Altenberg & Roberts, 2016;Bernstein Ratner & MacWhinney, 2016;Hadley, Rispoli, & Hsu, 2016;Long & Channell, 2001). Among these, mean length of utterance (MLU; Brown, 1973) has predominated; other such measures include Developmental Sentence Scoring (Lee, 1974), Assigning Structural Stage (Miller, 1981), and Language Assessment, Remediation and Screening (Crystal, Fletcher, & Garman, 1976).…”

“…It was presented primarily as a research tool, although with the hope "that other researchers and clinicians may find it helpful" (Scarborough, 1990, p. 13). Recent efforts have seen an increased focus on the value of the IPSyn for clinicians (e.g., Price et al, 2008), particularly in conjunction with computerized language sample analysis (LSA;Bernstein Ratner & MacWhinney, 2016;Long, 2001;Long & Channell, 2001).…”

“…When applicable, automation of scoring can dramatically reduce processing time and make it less labor intensive. Not surprisingly, therefore, there has been considerable interest in automated transcription and, regarding our focus, in how well IPSyn scoring might be accomplished by computers (Bernstein Ratner & MacWhinney, 2016;Hassanali et al, 2014;Long, 2001;Long & Channell, 2001;Sagae, Lavie, & MacWhinney, 2005). However, MacWhinney (2014) raised concerns that resemble those of our hand scorers, noting that "some of the IPSyn rules are difficult to interpret unambiguously" and that such ambiguity may pose difficulties for machine scoring.…”

Your Laptop to the Rescue: Using the Child Language Data Exchange System Archive and CLAN Utilities to Improve Child Language Sample Analysis

“…1 Some studies have estimated that it takes up to 8 hours of training and 45 minutes to an hour of work after a transcript has been generated to compute DSS. 10,11 One study estimated that it takes more than 30 minutes per sample following transcription to compute the Index of Productive Syntax (IPSyn). 12,13 It also seems to one of us, after a long career as a university instructor, that most LSA measures, even the time-honored MLU, are quite prone to error.…”