KOAS: Korean Text Offensiveness Analysis System

Abstract

Warning: This manuscript contains a certain level of offensive expression.As communication through social media platforms has grown immensely, the increasing prevalence of offensive language online has become a critical problem. Notably in Korea, one of the countries with the highest Internet usage, automatic detection of offensive expressions has recently been brought to attention. However, morphological richness and complex syntax of Korean causes difficulties in neural model training. Furthermore, most of previous studies mainly focus on the detection of abusive language, disregarding implicit offensiveness and underestimating a different degree of intensity. To tackle these problems, we present KOAS, a system that fully exploits both contextual and linguistic features and estimates an offensiveness score for a text. We carefully designed KOAS with a multi-task learning framework and constructed a Korean dataset for offensive analysis from various domains. Refer for a detailed demonstration. 1

Everyday life information seeking in South Korea during the COVID-19 pandemic: daily topics of information needs in social Q&A

“…Korean stop words were downloaded from three well-known sources and merged [1]. To find the nouns, the Open Korean Text part-of-speech tagger offered in the Korean NLP in Python module in Python (Park and Cho, 2014) was used. These preprocessed nouns were then converted into single- (unigrams) and double-word (bigrams) combinations for analysis.…”

Contents and sentiment analysis of newspaper articles and comments on telemedicine in Korea: Before and after of COVID-19 outbreak

“…It was demanding to manually investigate the plethora of text and comments of news articles, so natural language processing (NLP) procedures, including (1) tokenization, (2) stop words, and (3) stemming, were used in this study with the assistance of the Korean natural language processing in the Python (KoNLPy) package, 17,18 Korean natural language processing procedures were performed in a form that allows morphological analysis.…”

Graph Embedding-based Matching Multilingual Out-of-Vocabulary Terms on Social Media

“…Finally, use thirdparty word segmentation tools to perform word segmentation tasks. Our study choose Jieba [18] for Chinese segmentation, konlpy for Korean [19] and Sudachipy for Japanese [20].…”