IDENTIFYING SPEECH ACTS IN E‐MAILS: TOWARD AUTOMATED SCORING OF THE TOEIC^® E‐MAIL TASK

Abstract

Since its 1947 founding, ETS has conducted and disseminated scientific research to support its products and services, and to advance the measurement and education fields. In keeping with these goals, ETS is committed to making its research freely available to the professional community and to the general public. Published accounts of ETS research, including papers in the ETS Research Report series, undergo a formal peer-review process by ETS staff to ensure that they meet established scientific and professional standards. All such ETS-conducted peer reviews are in addition to any reviews that outside organizations may provide as part of their own publication processes. As part of its nonprofit mission, ETS conducts and disseminates the results of research to advance quality and equity in education and assessment for the benefit of ETS's constituents and the field.To obtain a PDF or a print copy of a report, please visit: AbstractThis study proposes an approach to automatically score the TOEIC ® Writing e-mail task. We focus on one component of the scoring rubric, which notes whether the test-takers have used particular speech acts such as requests, orders, or commitments. We developed a computational model for automated speech act identification and tested it on a corpus of TOEIC responses, achieving up to 79.28% accuracy. This model represents a positive first step toward the development of a more comprehensive scoring model. We also created a corpus of speech actannotated native English workplace e-mails. Comparisons between these and the TOEIC data allow us to assess whether English learners are approximating native models and whether differences between native and non-native data can have negative consequences in the global workplace.

Automated classification of collaborative problem solving interactions in simulated science tasks

“…This is lower than the 59.2% achieved when punctuation is used. It demonstrates that punctuation is clearly useful for classification, which is consistent with similar results in a different domain (De Felice and Deane, 2012).…”

Actionable Email Intent Modeling with Reparametrized RNNs

“…There has been a lot of work on multidomain/task learning with shared representation as we described in section 1. Our work is also closely related to work on email speech act modeling and recognition (Cohen, Carvalho, and Mitchell 2004;Lampert et al 2008;Jeong, Lin, and Lee 2009;De Felice and Deane 2012). The idea of model reparametrization for domain adaption is abundant in the literature of hierarchical Bayesian modeling, such as Finkel and Manning; Eisenstein, Ahmed, and Xing (2009; 2011).…”

Systematic community of Practice activities evaluation through Natural Language Processing: application to research projects

“…Pragmatics analysis of communication using e-mails uses only some of these methods like ngrams analysis by Carvalho and Cohen (2006), Verbal Response Mode scheme by Lampert in Lampert et al . (2010) or a custom coding scheme like in Felice and Deane (2012).…”