SpellCheF: Spelling Checker and Corrector for Filipino

Cheng, Charibeth; Alberto, Cedric Paul; Chan, Ian Anthony; Querol, Vazir Joshue

doi:10.3860/jrsce.v4i3.635

Cited by 7 publications

(4 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cross-lingual Character N-Gram Overlap (CROSSNGO). N-gram overlap has been used previously in various NLP tasks applied to Philippine language data such as language identification (Oco et al, 2013a;Cruz et al, 2016), spell checking and correction (Cheng et al, 2007;Octaviano and Borra, 2017;, and clustering (Oco et al, 2013b). Drawing inspiration from this fact and from the quantitative evidence of mutual intelligibility between Philippine languages presented in Section 2, we posit that a new feature designed specifically for closely related language data might improve the performance of the readability assess-ment models.…”

Section: Traditional Handcrafted Features (Trad) We Integrate Availab...mentioning

confidence: 99%

BERT Embeddings for Automatic Readability Assessment

Imperial¹

2021

Proceedings of the Conference Recent Advances in Natural Language Processing - Deep Learning for Natural Language Processing Me

View full text Add to dashboard Cite

In recent years, the main focus of research on automatic readability assessment (ARA) has shifted towards using expensive deep learningbased methods with the primary goal of increasing models' accuracy. This, however, is rarely applicable for low-resource languages where traditional handcrafted features are still widely used due to the lack of existing NLP tools to extract deeper linguistic representations. In this work, we take a step back from the technical component and focus on how linguistic aspects such as mutual intelligibility or degree of language relatedness can improve ARA in a low-resource setting. We collect short stories written in three languages in the Philippines -Tagalog, Bikol, and Cebuano -to train readability assessment models and explore the interaction of data and features in various crosslingual setups. Our results show that the inclusion of CROSSNGO, a novel specialized feature exploiting n-gram overlap applied to languages with high mutual intelligibility, significantly improves the performance of ARA models compared to the use of off-the-shelf large multilingual language models alone. Consequently, when both linguistic representations are combined, we achieve state-of-the-art results for Tagalog and Cebuano, and baseline scores for ARA in Bikol.

show abstract

Section: Traditional Handcrafted Features (Trad) We Integrate Availab...mentioning

confidence: 99%

BERT Embeddings for Automatic Readability Assessment

Imperial¹

2021

Proceedings of the Conference Recent Advances in Natural Language Processing - Deep Learning for Natural Language Processing Me

View full text Add to dashboard Cite

show abstract

“…Many approaches have been tried for word normalization in online Filipino text: (1) predetermined rules using commonly seen patterns (Guingab et al, 2014;Oco and Borra, 2011), (2) dictionary-substitution models for extracting patterns in misspelled words (Nocon et al, 2014), or (3) trigrams and Levenshtein or QWERTY distance to select words which share similar trigrams or are close in terms of edit or keyboard distance (Chan et al, 2008;Go et al, 2017).…”

Section: Related Workmentioning

confidence: 99%

“…Dictionary-substitution models are limited by the constraint of picking mapping each pattern to only a single substitution, whereas in reality, dif-ferent patterns may need to be applied to different words bearing the same pattern (Nocon et al, 2014). Trigrams and distance metrics alone may be successful in the context of correcting typographical errors for which the model was developed (Chan et al, 2008), but may not be as successful on intentionally abbreviated words. Our work uses a combination of these methods to develop a model that can be easily updated, considers multiple possible candidates, and works in the online text setting.…”

Section: Related Workmentioning

confidence: 99%

Look Ma, Only 400 Samples! Revisiting the Effectiveness of Automatic N-Gram Rule Generation for Spelling Normalization in Filipino

Flores¹,

Radev²

2022

Proceedings of the Third Workshop on Simple and Efficient Natural Language Processing (SustaiNLP)

View full text Add to dashboard Cite

With 84.75 million Filipinos online, the ability for models to process online text is crucial for developing Filipino NLP applications. To this end, spelling correction is a crucial preprocessing step for downstream processing. However, the lack of data prevents the use of language models for this task. In this paper, we propose an N-Gram + Damerau-Levenshtein distance model with automatic rule extraction. We train the model on 300 samples, and show that despite limited training data, it achieves good performance and outperforms other deep learning approaches in terms of accuracy and edit distance. Moreover, the model (1) requires little compute power, (2) trains in little time, thus allowing for retraining, and (3) is easily interpretable, allowing for direct troubleshooting, highlighting the success of traditional approaches over more complex deep learning models in settings where data is unavailable.

show abstract

“…Grammar checkers are some of the applications where syntactic specification of languages is necessary. SpellCheF is a spell checker for Filipino that uses a hybrid approach in detecting and correcting misspelled words in a document (Cheng, et al, 2007). Its approach is composed of dictionary-lookup, n-gram analysis, Soundex and character distance measurements.…”

Section: Language Grammarsmentioning

confidence: 99%