The Japanese lexical transducer based on stem-suffix style forms

Abstract

A Lexical Transducer (LT) as defined by Karttunen, Kaplan, Zaenen 1992 is a specialized finite state transducer (FST) that relates citation forms of words and their morphological categories to inflected surface forms. Using LTs is advantageous because the same structure and algorithms can be used for morphological analysis (stemming) and generation. Morphological processing (analysis and generation) is computationally faster, and the data for the process can be compacted more tightly than with other methods. The standard way to construct an LT consists of three steps: (1) constructing a simple finite state source lexicon LA which defines all valid canonical citation forms of the language; (2) describing morphological alternations by means of two-level rules, compiling the rules to FSTs, and intersecting them to form a single rule transducer RT; and (3) composing LA and RT.

Deep Sentiment Analysis: A Case Study on Stemmed Turkish Twitter Data

“…The stemming rather reduces the information gained from the data in many languages. In fact, the stemming improves accuracy (ACC [28]) achieved by various methods in different languages including not only English [29], [30] but also Arabic [26], [27], [31], [32], Indonesian [23], [33], [34], Japanese [25], [35] French [36]- [38], Portuguese [37], [39], German [37], [40], [41], Hungarian [37], [42], [43], Spanish [44]- [47], and Turkish [48]- [50].…”