Word Segmentation of Informal Arabic with Domain Adaptation

Monroe, Will; Green, Spence; Manning, Christopher D.

doi:10.3115/v1/p14-2034

Cited by 59 publications

(45 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This adaptation scheme is attractive because of its simplicity and ease of use as a pre-processing step, and also because it performs quite well despite its simplicity. It has been used in various NLP tasks such as word segmentation (Monroe et al, 2014), machine translation , word sense disambiguation (Zhong et al, 2008), and short answer scoring (Heilman and Madnani, 2013). Our work is an extension of this scheme in the sense that our work is a generalization of EasyAdapt.…”

Section: Domain Adaptationmentioning

confidence: 99%

Flexible Domain Adaptation for Automated Essay Scoring Using Correlated Linear Regression

Phandi¹,

Chai²,

Ng³

2015

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing

106

138

View full text Add to dashboard Cite

Most of the current automated essay scoring (AES) systems are trained using manually graded essays from a specific prompt. These systems experience a drop in accuracy when used to grade an essay from a different prompt. Obtaining a large number of manually graded essays each time a new prompt is introduced is costly and not viable. We propose domain adaptation as a solution to adapt an AES system from an initial prompt to a new prompt. We also propose a novel domain adaptation technique that uses Bayesian linear ridge regression. We evaluate our domain adaptation technique on the publicly available Automated Student Assessment Prize (ASAP) dataset and show that our proposed technique is a competitive default domain adaptation algorithm for the AES task.

show abstract

Section: Domain Adaptationmentioning

confidence: 99%

Flexible Domain Adaptation for Automated Essay Scoring Using Correlated Linear Regression

Phandi¹,

Chai²,

Ng³

2015

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing

106

138

View full text Add to dashboard Cite

show abstract

“…For this task, we utilize the Stanford Word Segmenter (Monroe et al 2014). 19 The Arabic texts were POS tagged and parsed using the Stanford Arabic Parser.…”

Section: Arabicmentioning

confidence: 99%

Multilingual native language identification

Malmasi¹,

Dras²

2015

Nat. Lang. Eng.

View full text Add to dashboard Cite

We present the first comprehensive study of Native Language Identification (NLI) applied to text written in languages other than English, using data from six languages. NLI is the task of predicting an author's first language using only their writings in a second language, with applications in Second Language Acquisition and forensic linguistics. Most research to date has focused on English but there is a need to apply NLI to other languages, not only to gauge its applicability but also to aid in teaching research for other emerging languages. With this goal, we identify six typologically very different sources of non-English second language data and conduct six experiments using a set of commonly used features. Our first two experiments evaluate our features and corpora, showing that the features perform well and at similar rates across languages. The third experiment compares non-native and native control data, showing that they can be discerned with 95 per cent accuracy. Our fourth experiment provides a cross-linguistic assessment of how the degree of syntactic data encoded in partof-speech tags affects their efficiency as classification features, finding that most differences between first language groups lie in the ordering of the most basic word categories. We also tackle two questions that have not previously been addressed for NLI. Other work in NLI has shown that ensembles of classifiers over feature types work well and in our final experiment we use such an oracle classifier to derive an upper limit for classification accuracy with our feature set. We also present an analysis examining feature diversity, aiming to estimate the degree of overlap and complementarity between our chosen features employing an association measure for binary data. Finally, we conclude with a general discussion and outline directions for future work.

show abstract

“…D is the same baseline as Green et al 5 We tokenized the English with Stanford CoreNLP according to the Penn Treebank standard (Marcus et al, 1993), the Arabic with the Stanford Arabic segmenter (Monroe et al, 2014) according to the Penn Arabic Treebank standard (Maamouri et al, 2008), and the Chinese with the Stanford Chinese segmenter (Chang et al, 2008) according to the Penn Chinese Treebank standard (Xue et al, 2005). 6 Data sources: tune, MT023568; dev, MT04; dev-dom, domain adaptation dev set is MT04 and all wb and bn data from LDC2007E61; test1, MT09 (Ar-En) and MT12 (Zh-En); test2, Progress0809 which was revealed in the OpenMT 2012 evaluation; test3, MetricsMATR08-10.…”

Section: Resultsmentioning

confidence: 99%

An Empirical Comparison of Features and Tuning for Phrase-based Machine Translation

Green¹,

Cer²,

Manning³

2014

Proceedings of the Ninth Workshop on Statistical Machine Translation

Self Cite

View full text Add to dashboard Cite

Scalable discriminative training methods are now broadly available for estimating phrase-based, feature-rich translation models. However, the sparse feature sets typically appearing in research evaluations are less attractive than standard dense features such as language and translation model probabilities: they often overfit, do not generalize, or require complex and slow feature extractors. This paper introduces extended features, which are more specific than dense features yet more general than lexicalized sparse features. Large-scale experiments show that extended features yield robust BLEU gains for both Arabic-English (+1.05) and Chinese-English (+0.67) relative to a strong feature-rich baseline. We also specialize the feature set to specific data domains, identify an objective function that is less prone to overfitting, and release fast, scalable, and language-independent tools for implementing the features.

show abstract

Word Segmentation of Informal Arabic with Domain Adaptation

Cited by 59 publications

References 7 publications

Flexible Domain Adaptation for Automated Essay Scoring Using Correlated Linear Regression

Flexible Domain Adaptation for Automated Essay Scoring Using Correlated Linear Regression

Multilingual native language identification

An Empirical Comparison of Features and Tuning for Phrase-based Machine Translation

Contact Info

Product

Resources

About