Text analysis and language identification for polyglot text-to-speech synthesis

Lui²,

Zampieri³

et al. 2019

jair

102

Language identification ("LI") is the problem of determining the natural language that a document or part thereof is written in. Automatic LI has been extensively researched for over fifty years. Today, LI is a key part of many text processing pipelines, as text processing techniques generally assume that the language of the input text is known. Research in this area has recently been especially active. This article provides a brief history of LI research, and an extensive survey of the features and methods used in the LI literature. We describe the features and methods using a unified notation, to make the relationships between methods clearer. We discuss evaluation methods, applications of LI, as well as off-the-shelf LI systems that do not require training by the end user. Finally, we identify open issues, survey the work to date on each issue, and propose future directions for research in LI.LI as a task predates computational methods -the earliest interest in the area was motivated by the needs of translators, and simple manual methods were developed to quickly identify documents in specific languages. The earliest known work to describe a functional LI program for text is by Mustonen (1965), a statistician, who used multiple discriminant analysis to teach a computer how to distinguish, at the word level, between English, Swedish and Finnish. Mustonen compiled a list of linguistically-motivated character-based features, and trained his language identifier on 300 words for each of the three target languages. The training procedure created two discriminant functions, which were tested with 100 words for each language. The experiment resulted in 76% of the words being correctly classified; even by current standards this percentage would be seen as acceptable given the small amount of training material, although the composition of training and test data is not clear, making the experiment unreproducible.In the early 1970s, Nakamura (1971) considered the problem of automatic LI. According to Rau (1974) and the available abstract of Nakamura's article, 1 his language identifier was able to distinguish between 25 languages written with the Latin alphabet. As features, the method used the occurrence rates of characters and words in each language. From the abstract it seems that, in addition to the frequencies, he used some binary presence/absence features of particular characters or words, based on manual LI. Rau (1974) wrote his master's thesis "Language Identification by Statistical Analysis" for the Naval Postgraduate School at Monterey, California. The continued interest and the need to use LI of text in military intelligence settings is evidenced by the recent articles of, for example, Rafidha Rehiman et al. (2013), Rowe et al. (2013), and Voss et al. (2014. As features for LI, Rau (1974) used, e.g., the relative frequencies of characters and character bigrams. With a majority vote classifier ensemble of seven classifiers using Kolmogor-Smirnov's Test of Goodness of Fit and Yule's characteristic (K), he managed...

Section: Morphemes Syllables and Chunksmentioning

confidence: 99%

Section: Statistics Of Words Van Der Lee and Boschmentioning

confidence: 99%

Section: Co-occurrences Of Wordsmentioning

confidence: 99%

See 1 more Smart Citation

Automatic Language Identification in Texts: A Survey

Lui²,

Zampieri³

et al. 2019

jair

102

“…We triangulated the qualitative data with descriptive statistics of the five most frequent word stems in the psychotherapists’ responses from each country to give a numerical impression of the most frequent responses. To this end, the entire text corpus was analysed to identify the most frequently used word stems using the word stem analysis method [ 49 , 50 ]. A total of 4293 word stems were identified and manually checked to avoid possible different semantic meanings of the word stems in the individual testimonies.…”

Section: Methodsmentioning

confidence: 99%

Psychotherapists’ Reports regarding the Impact of the COVID-19 Pandemic on Their Patients: A Cross-National Descriptive Study Based on the Social-Ecological Model (SEM)

Schaffler

Kuška

Barke

et al. 2022

IJERPH

The COVID-19 outbreak has raised questions about how vulnerable groups experience the pandemic. Research that focuses on the view of individuals with pre-existing mental health conditions is still limited, and so are cross-country comparative surveys. We gathered our sample of qualitative data during the first lockdown after governmental measures against the spread of the SARS-CoV-2 virus came into force in Austria, Czechia, Germany, and Slovakia. A total of n = 1690 psychotherapists from four middle European countries answered the question of how the COVID-19 pandemic was addressed in sessions by their patients during the early stage of unprecedented public health conditions. We employed a descriptive qualitative methodology to determine themes following levels of the social-ecological model (SEM) regarding how the COVID-19 pandemic affected patients. At the public policy level, stressful environmental conditions concerned the governmental mitigation efforts. At the level of community/society, reported key themes were employment, restricted access to educational and health facilities, socioeconomic consequences, and the pandemic itself. Key themes at the interpersonal level regarded forced proximity, the possibility of infection of loved ones, childcare, and homeschooling. Key themes at the individual level were the possibility of contracting COVID-19, having to stay at home/isolation, and a changing environment. Within the SEM framework, adaptive and maladaptive responses to these stressors were reported, with more similarities than differences between the countries. A quantification of word stems showed that the maladaptive reactions predominated.

“…They evaluated their method with single sentences in 9 languages. Romsdorfer considered language identification with multilingual text-to-speech synthesis [15] [16]. In 2010, Murthy and Kumar [7] classified small text samples between two Indian languages.…”

Section: Earlier Workmentioning

confidence: 99%

Language Set Identification in Noisy Synthetic Multilingual Documents

Lindén

Computational Linguistics and Intelligent Text Processing

2015

Abstract. In this paper, we reconsider the problem of language identification of multilingual documents. Automated language identification algorithms have been improving steadily from the seventies until recent years. The current state-of-the-art language identifiers are quite efficient even with only a few characters and this gives us enough reason to again evaluate the possibility to use existing language identifiers for monolingual text to detect the language set of a multilingual document. We are using a previously developed language identifier for monolingual documents with the multilingual documents from the WikipediaMulti dataset published in a recent study. Our method outperforms previous methods tested with the same data, achieving an F1-score of 97.6 when classifying between 44 languages.