A Report on the Third

Zampieri, Marcos; Malmasi, Shervin; Scherrer, Yves; Samardžić, Tanja; Tyers, Francis M.; Silfverberg, Miikka; Klyueva, Natalia; Pan, Tung-Le; Huang, Chu‐Ren; Ionescu, Radu Tudor; Butnaru, Andrei M.; Jauhiainen, Tommi

doi:10.18653/v1/w19-1401

Cited by 15 publications

(15 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This tendency is also found in other related tasks such as discriminating between similar languages and varieties. In the last VarDial Evaluation Campaign [24], as in previous years, systems based on neural networks did not reach competitive scores.…”

Section: Corpus-based Approachesmentioning

confidence: 73%

Measuring Language Distance of Isolated European Languages

Gamallo

Campos²,

Alegria

2020

Information

View full text Add to dashboard Cite

Phylogenetics is a sub-field of historical linguistics whose aim is to classify a group of languages by considering their distances within a rooted tree that stands for their historical evolution. A few European languages do not belong to the Indo-European family or are otherwise isolated in the European rooted tree. Although it is not possible to establish phylogenetic links using basic strategies, it is possible to calculate the distances between these isolated languages and the rest using simple corpus-based techniques and natural language processing methods. The objective of this article is to select some isolated languages and measure the distance between them and from the other European languages, so as to shed light on the linguistic distances and proximities of these controversial languages without considering phylogenetic issues. The experiments were carried out with 40 European languages including six languages that are isolated in their corresponding families: Albanian, Armenian, Basque, Georgian, Greek, and Hungarian.

show abstract

Section: Corpus-based Approachesmentioning

confidence: 73%

Measuring Language Distance of Isolated European Languages

Gamallo

Campos²,

Alegria

2020

Information

View full text Add to dashboard Cite

show abstract

“…As of this writing, the results from the shared tasks of VarDial 2019 (Zampieri et al . 2019) are being prepared for publication, and several participating teams had incorporated some sort of an LM adaptation algorithm into their systems. We used the same LM adaptation scheme as presented in this paper with the HeLI 2.0 method as well as with a custom NB implementation (Jauhiainen, Jauhiainen, and Lindén 2019), two teams used such a scheme with SVMs (Benites, von Däniken, and Cieliebak 2019; Wu et al .…”

Section: Discussionmentioning

confidence: 99%

“…We were delighted to see that some of the other participants of the 2019 VarDial Evaluation Campaign (Zampieri et al . 2019) had noticed our LM adaptation scheme and used a somewhat similar way of gathering new information from the test sets with their own systems.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Language model adaptation for language and dialect identification of text

2019

View full text Add to dashboard Cite

This article describes an unsupervised language model adaptation approach that can be used to enhance the performance of language identification methods. The approach is applied to a current version of the HeLI language identification method, which is now called HeLI 2.0. We describe the HeLI 2.0 method in detail. The resulting system is evaluated using the datasets from the German dialect identification and Indo-Aryan language identification shared tasks of the VarDial workshops 2017 and 2018. The new approach with language identification provides considerably higher F1scores than the previous HeLI method or the other systems which participated in the shared tasks. The results indicate that unsupervised language model adaptation should be considered as an option in all language identification tasks, especially in those where encountering out-of-domain data is likely.

show abstract

“…On the other hand, using orthography will render the task trivial and irrelevant as a linguistic study. Thus, we do convert both sets of data to the same orthography convention, just like the DMT task reported in Zampieri et al (2019). This setup means that our study will be among the first to focus on identification and classification of similar languages with minimal conventionalized linguistic cues.…”

Section: Introductionmentioning

confidence: 99%

Classification of regional and genre varieties of Chinese: A correspondence analysis approach based on comparable balanced corpora

Hou

Huang

2020

Nat. Lang. Eng.

Self Cite

View full text Add to dashboard Cite

This paper proposes a robust text classification and correspondence analysis approach to identification of similar languages. In particular, we propose to use the readily available information of clauses and word length distribution to model similar languages. The modeling and classification are based on the hypothesis that languages are self-adaptive complex systems and hence can be classified by dynamic features describing the system, especially in terms of distributional relations of constituents of a system. For similar languages whose grammatical differences are often subtle, classification based on dynamic system features should be more effective. To test this hypothesis, we considered both regional and genre varieties of Mandarin Chinese for classification. The data are extracted from two comparable balanced corpora to minimize possible confounding factors. The two corpora are the Sinica Corpus from Taiwan and the Lancaster Corpus of Mandarin Chinese from Mainland China, and the two genres are reportage and review. Our text classification and correspondence analysis results show that the linguistically felicitous two-level constituency model combining power functions between word and clauses effectively classifies the two varieties of Chinese for both genres. In addition, we found that genres do have compounding effect on classification of regional varieties. In particular, reportage in two varieties is more likely to be classified than review, corroborating the complex system view of language variations. That is, language variations and changes typically do not take place evenly across the board for the complete language system. This further enhances our hypothesis that dynamic complex system features, such as the power functions captured by the Menzerath–Altmann law, provide effective models in classifications of similar languages.

show abstract

A Report on the Third

Cited by 15 publications

References 41 publications

Measuring Language Distance of Isolated European Languages

Measuring Language Distance of Isolated European Languages

Language model adaptation for language and dialect identification of text

Classification of regional and genre varieties of Chinese: A correspondence analysis approach based on comparable balanced corpora

Contact Info

Product

Resources

About