Challenges Encountered in Turkish Natural Language Processing Studies

Tohma, Kadir; Kutlu, Yakup

doi:10.28978/nesciences.833188

Cited by 4 publications

(1 citation statement)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Özellikle sondan eklemeli bir dil yapısına sahip olması, ünlü-ünsüz uyumu v.b. birçok farklı özelliğe sahip olması sebebiyle dil tanıma konusunda araştırmacılar için ilginç bir yönü olduğuna değinmişlerdir [27]. Tian ve Suontausta'ın yaptıkları çalışmada sinir ağı tabanlı ölçeklenebilir yeni bir dil tanımlaması yöntemi önermişlerdir.…”

Section: Li̇teratür çAlişmalari (Literature Studies)unclassified

Döküman dili tanıma için içerik bağımsız yeni bir yaklaşım: Açı Örüntüler

NOYAN

KUNCAN

Tekin

et al. 2022

Gazi Üniversitesi Mühendislik Mimarlık Fakültesi Dergisi

View full text Add to dashboard Cite

 Language Identification in text mining is the process of detecting the natural language in which a document or part of it is written  The proposed angle pattern method is used for feature extraction from texts  The performance results of the proposed language recognition method in four different data sets are between 93,31% and 99,39%Language identification (LI) in text mining is the process of detecting the natural language in which a document or part of it is written. LI aims to mimic a human's ability to recognize certain languages from text by computer algorithms. LI can be defined as a classification problem subject based on the information used in word or character size for any document. When the literature is examined for LI application, it is seen that various linguistic or statistical-based approaches are used. Linguistic methods are methods that perform LI according to a special word or character of a language. These methods are applied based on the special rules of the languages. When we look at the statistical methods, it shows that the words or characters that make up the language depend on their frequency and distribution. The statistical approaches used are contentindependent methods. The semantic context of the text is not concerned with its content. According to linguistic methods, it does not provide sufficient information about the content of the text. The proposed model in this study is a statistical approach. Figure A. Proposed block diagram for LI Purpose: In this study, a new LI approach using the angle information between the UTF-8 values of the characters in the text is proposed. The proposed angle pattern method is used for feature extraction from texts.Angle patterns method is a statistical approach. In the angle method, there are two distance parameters, R and L, which express which neighborhood to look at from the reference point to the left and right. Theory and Methods:To test the proposed approach, four datasets, two created by the authors and two publicly available on the Internet, were used. By using the features obtained by the angle pattern method, classification process was carried out with different machine learning methods such as Random Forest, Support Vector Machine, Linear Discriminant Analysis, Naive Bayes and K-nearest neighbor. Language identification performance results determined from four different data sets were observed as 96.81%, 99.39%, 93.31% and 98.60%, respectively. Results:According to the performance results achieved as a result of the study, it has been determined that the proposed angle pattern method provides important distinguishing information in language identification application. It is thought that the proposed approach in this study can be used in many different text mining applications such as spam recognition, text categorization, as well as LI application.

show abstract

Section: Li̇teratür çAlişmalari (Literature Studies)unclassified