Developing and testing syllabification systems for South African Sesotho

Abstract

In this article, we describe the development of two syllabification systems for South African Sesotho. First, we implemented a rule-based syllabification system based on the syllabification rules proposed by Guma (An outline structure of Southern Sotho, 2nd edn, Shooter and Shuter Publishers, Pietermaritzburg, 1982). The rules describe the location of syllable boundaries based on the location of vowels, consonants, simple and complex nasal consonants, and the lateral l. Second, we used TeX’s pattern-based hyphenation system as a syllabification system. This pattern-based system was trained on syllabified words from a dictionary as identified in Chitja (Phatlamantsoe ya Sesotho ya Machaba, Mazenod Publishers, Lesotho, 2010). The syllable boundaries in the words from the dictionary were manually checked and, where needed, adjusted to maintain a consistent orthography. Both systems were evaluated on 13,551 tokens using ten-fold cross-validation (as the pattern-based system requires training data). The rule-based system shows a 99.69% word level accuracy, which indicates that Sesotho syllabification is highly regular. The pattern-based system leads to 78.97% accuracy. This lower score is likely related to the limitations in the placement of boundaries in the TeX system. A qualitative analysis of the results shows some (unresolved) inconsistencies in the dictionary entries, in particular related to the conversion of the Lesotho Sesotho orthography to the South African Sesotho orthography and the handling of loan words.