A novel channel distortion measure for vector quantization and a fuzzy model for codebook index assignment

Abstract

This paper describes a methodology for semi-automatic grammar induction from unannotated corpora belonging to a restricted domain. The grammar contains both semantic and syntactic structures, which are conducive towards language understanding. Our work aims to ameliorate the reliance of grammar development on expert handcrafting or the availability of annotated corpora. To strive for a reasonable model for real data, as well as portability across domain and languages, we adopt a statistical approach. Our approach is also amenable to the optional injection of prior knowledge to aid grammar induction, and subsequent hand editing for grammar refinement. This constitutes the semi-automatic nature of the approach. Experiments with the ATIS corpus showed positive results in semantic parsing, when compared to an entirely handcrafted grammar.

Example-based bi-directional Chinese-English machine translation with semi-automatically induced grammars

“…This work extends our previous effort in the use of semiautomatically induced grammars for bi-directional English-Chinese machine translation using an example-based approach [1,2]. Our parallel experimental corpora includes the English ATIS-3 Class A sentences (training set, test set 1993 and 1994) with their Chinese translations.…”

“…We ran our grammar induction procedure with the distance metrics KL, MN and GI to generate three grammars respectively -G KL , G MN and G GI . 2 We compared these three grammars at every tenth iteration until the stopping criterion is met. 3 When we evaluated with the ATIS training set and rank the grammars in decreasing order of precision (P), we observed (G MN > G GI > G KL ) across the various iterations.…”

“…The BLEU (Bilingual Evaluation Understudy) metric was recently proposed by IBM [8] for automatic evaluation of machine translation. BLEU compares variable length phrases of the translated result against multiple 2 All other experimental parameters are controlled. No.…”

“…In spatial clustering, words or multi-word entities with similar left and right linguistic contexts are clustered together based on the symmetrized divergence (Div) that is applied to the left and right linguistic contexts of the entity pair. Div incorporates the Kullback-Leibler (KL) distance metric [1,2] (See Equation 1). 1 In general, temporal clustering generates phrasal categories in the grammar and spatial clustering generates semantic categories.…”

Automatic concept identification in goal-oriented conversations

“…Acquiring domain information is a resource intensive effort but is a necessary part of making language technologies useful. Automatic techniques have been described for language modeling [1,2] and as an adjunct to grammar writing [3,4,5] for spoken language systems. Parallel efforts, though with somewhat different goals and approaches, exist in text processing [6].…”