KLUE: Korean Language Understanding Evaluation

Park, Sungjoon; Moon, Jihyung; Kim, Sungdong; Cho, Won Ik; Han, Ji Yoon; Park, Jangwon; Song, Chisung; Kim, Junseong; Song, Youngsook; Oh, Taehwan; Lee, Joohong; Oh, Juhyun; Lyu, Sungwon; Jeong, Younghoon; Lee, Inkwon; Lee, Dongjun; Kim, Hyun Woo; Lee, Myeonghwa; Jang, Seongbo; Do, Seungwon; Kim, Sunkyoung; Lim, KyungTae; Lee, Jongwon; Park, Kyumin; Shin, Jamin; Kim, Seonghyun; Park, Eunjeong Lucy; Oh, Alice; Ha, Jung-Woo; Cho, Kyunghyun

doi:10.48550/arxiv.2105.09680

Cited by 12 publications

(22 citation statements)

References 91 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The baseline for this study was a model trained on open-source datasets during both training and evaluation processes. The dataset used for the large classification baseline was the KLUE-NER dataset (Korean Language Understanding Evaluation Dataset for Named Entity Recognition) [24], which is a massive Korean dataset constructed for named " as "gender", and "50 In token classification tasks, when the text "The defendant lived with the victim, Ms. Lee (female, 50 years old)…" is input, the model first splits the entire text based on spaces and learns the role of each token within sentences or paragraphs and its relationship with surrounding words. Then, the model calculates the probability of each token being key information and predicts the highest-probability key information.…”

Section: Experimental Results Of Benchmark Modelsmentioning

confidence: 99%

Section: Experimental Results Of Benchmark Modelsmentioning

confidence: 99%

See 1 more Smart Citation

Key Information Extraction for Crime Investigation by Hybrid Classification Model

Park,

Kim

2024

Electronics

View full text Add to dashboard Cite

The 2021 amendment to South Korea’s Criminal Procedure Law has significantly enhanced the role of the police as investigative authorities. Consequently, there is a heightened demand for advanced investigative expertise among the police, driven by an increase in the number of cases each investigator handles and the extended time required for report preparation. This situation underscores the necessity for an artificial-intelligence-supported system to augment the efficiency of investigators. In response, this study designs a hybrid model that fine-tunes two Transformer-based pre-trained language models to automatically extract 18 key pieces of information from legal documents. To facilitate this, “The Major Information Frame of Homicide Criminal Facts” was developed, and a large-scale training dataset specialized in the criminal investigation field was constructed. The hybrid classification model proposed in this research achieved an F1 score of 87.75%, indicating superior performance compared to using a single machine reading model. Additionally, the model’s top three predicted answers included the correct answer at a rate exceeding 98%, demonstrating a high accuracy level. These results suggest that the hybrid classification model designed in this study can play a crucial role in efficiently extracting essential information from complex legal and investigative documents. Based on these findings, it is confirmed that the hybrid classification model can be applied not only in drafting investigative reports but also in tasks such as searching for similar case precedents and constructing case timelines in various legal and investigative applications. The advancement is expected to provide a standardized approach that allows all investigators to perform objective investigations and hypothesis testing, thereby enhancing the fairness and efficiency of the investigative process.

show abstract

Section: Experimental Results Of Benchmark Modelsmentioning

confidence: 99%

Section: Experimental Results Of Benchmark Modelsmentioning

confidence: 99%

Key Information Extraction for Crime Investigation by Hybrid Classification Model

Park,

Kim

2024

Electronics

View full text Add to dashboard Cite

show abstract

“…We choose KLUE-BERT-base [ 14 ], KoElectra [ 15 ], KorSciBERT and KorSciElctra ( (accessed on 30 December 2022)) as pre-trained language models to add extension vocabulary modules. We deliver the output of the final layer for the first input token to the linear layer to predict.…”

Section: Resultsmentioning

confidence: 99%

Domain Word Extension Using Curriculum Learning

Seong

Cha

2023

Sensors

View full text Add to dashboard Cite

Self-supervised learning models, such as BERT, have improved the performance of various tasks in natural language processing. Although the effect is reduced in the out-of-domain field and not the the trained domain thus representing a limitation, it is difficult to train a new language model for a specific domain since it is both time-consuming and requires large amounts of data. We propose a method to quickly and effectively apply the pre-trained language models trained in the general domain to a specific domain’s vocabulary without re-training. An extended vocabulary list is obtained by extracting a meaningful wordpiece from the training data of the downstream task. We introduce curriculum learning, training the models with two successive updates, to adapt the embedding value of the new vocabulary. It is convenient to apply because all training of the models for downstream tasks are performed in one run. To confirm the effectiveness of the proposed method, we conducted experiments on AIDA-SC, AIDA-FC, and KLUE-TC, which are Korean classification tasks, and subsequently achieved stable performance improvement.

show abstract

“…There were a few SQuAD format datasets released in non-English languages. Some examples are KorQuAD 1.0 [27], KorQuAD 2.0 [26], KLUE-MRC [15], FQuAD 1.1 [6], GermanQuAD [13], and SberQuAD [7]. Ko-rQuAD 1.0 is a Korean QA dataset that contains over 70k samples.…”

Section: Reading Comprehension In Other Languagesmentioning

confidence: 99%

“…Computing F1 in words is not trivial in Japanese because Japanese sentences do not have spaces. We chose a character-level F1 score as an evaluation metric by referring to the use of character-based evaluation metrics in Korean QA datasets [27,26,15]. Because Japanese uses thousands of kanji (Chinese characters) and each kanji has a meaning, the probability of two phrases coincidentally overlapping by character is low when the two phrases have different meanings.…”

Section: Dataset Evaluationmentioning

confidence: 99%

JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension

So¹,

Byun²,

Kang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Question Answering (QA) is a task in which a machine understands a given document and a question to find an answer. Despite impressive progress in the NLP area, QA is still a challenging problem, especially for non-English languages due to the lack of annotated datasets. In this paper, we present the Japanese Question Answering Dataset, JaQuAD, which is annotated by humans. JaQuAD consists of 39,696 extractive question-answer pairs on Japanese Wikipedia articles. We finetuned a baseline model which achieves 78.92% for F1 score and 63.38% for EM on test set. The dataset and our experiments are available at https://github.com/SkelterLabsInc/JaQuAD.

show abstract

KLUE: Korean Language Understanding Evaluation

Abstract: We introduce Korean Language Understanding Evaluation (KLUE) benchmark. KLUE is a collection of 8 Korean natural language understanding (NLU)

Cited by 12 publications

References 91 publications

Key Information Extraction for Crime Investigation by Hybrid Classification Model

Key Information Extraction for Crime Investigation by Hybrid Classification Model

Domain Word Extension Using Curriculum Learning

JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension

Contact Info

Product

Resources

About