Statistical Machine Translation Approach for Lexical Normalization on Indonesian Text

Kurnia, Ajmal; Yulianti, Evi

doi:10.1109/ialp51396.2020.9310508

Cited by 2 publications

(3 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…MT stage. This idea is supported by previous work [28], [29], [30]. Veliz et al [28] found that a ruleset could be used for lexical normalisation as a pre-normalisation step before the text was processed using MT.…”

Section: A Code-mixed Normalisationmentioning

confidence: 52%

“…Veliz et al [28] found that a ruleset could be used for lexical normalisation as a pre-normalisation step before the text was processed using MT. Kurnia and Yulianti [29] confirmed that a ruleset could be applied and proved to be useful for lexical normalisation. Yulianti et al [30] applied rule-based MT (RBMT) before the text was inputted into statistical MT (SMT), and showed that the resulting hybrid MT was more effective than using RBMT or SMT alone.…”

Section: A Code-mixed Normalisationmentioning

confidence: 79%

See 1 more Smart Citation

Normalisation of Indonesian-English Code-Mixed Text and its Effect on Emotion Classification

Yulianti¹,

Kurnia²,

Adriani³

et al. 2021

IJACSA

Self Cite

View full text Add to dashboard Cite

Usage of code-mixed text has increased in recent years among Indonesian internet users, who often mix Indonesian-language with English-language text. Normalisation of this code-mixed text into Indonesian needs to be performed to capture the meaning of English parts of the text and process them effectively. We improve a state-of-the-art code-mixed Indonesian-English normalisation system by modifying its pipeline modules. We further analyse the effect of code-mixed normalisation on emotion classification tasks. Our approach significantly improved on a state-of-the-art Indonesian-English code-mixed text normalisation system in both the individual pipeline modules and the overall system. The new feature set in the language identification module showed an improvement of 4.26% in terms of F 1 score. The combination of machine translation and ruleset in the lexical normalisation module improved BLEU score by 25.22% and lowered WER by 62.49%. The use of context in the translation module improved BLEU score by 2.5% and lowered WER by 8.84%. The effectiveness of the overall pipeline normalisation system increased by 32.11% and 33.82%, in terms of BLEU score and WER, respectively. Code-mixed normalisation also improved the accuracy of emotion classification by up to 37.74% in terms of F 1 score.

show abstract

Section: A Code-mixed Normalisationmentioning

confidence: 52%

Section: A Code-mixed Normalisationmentioning

confidence: 79%

Normalisation of Indonesian-English Code-Mixed Text and its Effect on Emotion Classification

Yulianti¹,

Kurnia²,

Adriani³

et al. 2021

IJACSA

Self Cite

View full text Add to dashboard Cite

show abstract

“…Data normalization is used to make improvements to nonstandard words or abbreviated words. This improvement uses a normalized dictionary that matches the standard word characters in Indonesian [22]. An example of the normalized dictionary used in this study can be seen in Table 4.…”

Section: Data Normalizationmentioning

confidence: 99%

Application of LSTM and GloVe Word Embedding for Hate Speech Detection in Indonesian Twitter Data

Imaduddin,

Kusumaningtias,

A'la

2023

ISI

View full text Add to dashboard Cite

Hate speech, characterized by intentional expressions of dissatisfaction, is a prevalent phenomenon on social media platforms, including Twitter. Its continual occurrence can foster divisions, misunderstandings, and even acts of violence between individuals and groups, particularly due to the resulting prejudice. This study investigates the occurrence of hate speech within Indonesian content on Twitter, employing a deep learning approach to detect and analyze such expressions. The Long Short-Term Memory (LSTM) method, coupled with the GloVe word embedding technique, is utilized on a dataset comprising 13,169 Indonesian tweets flagged for hate speech. Four distinct model architectures were developed through the integration of LSTM and GloVe. The findings reveal model 1 to exhibit superior performance, achieving a precision of 89%, a recall of 99%, an F-1 score of 94%, and an overall accuracy of 94.24%. It is suggested that future research explore the potential deployment of this model in web or mobile platforms for real-time analysis, thereby enhancing the capacity for immediate hate speech detection and mitigation.

show abstract

Statistical Machine Translation Approach for Lexical Normalization on Indonesian Text

Cited by 2 publications

References 19 publications

Normalisation of Indonesian-English Code-Mixed Text and its Effect on Emotion Classification

Normalisation of Indonesian-English Code-Mixed Text and its Effect on Emotion Classification

Application of LSTM and GloVe Word Embedding for Hate Speech Detection in Indonesian Twitter Data

Contact Info

Product

Resources

About