This work lays the foundation for automated assessments of narrative quality in student writing. We first manually score essays for narrative-relevant traits and sub-traits, and measure inter-annotator agreement. We then explore linguistic features that are indicative of good narrative writing and use them to build an automated scoring system. Experiments show that our features are more effective in scoring specific aspects of narrative quality than a state-of-the-art feature set.

The literature and the employee and workforce surveys rank collaborative problem solving (CPS) among the top 5 most critical skills necessary for success in college and the workforce. This paper provides a review of the literature on CPS and related terms, including a discussion of their definitions, importance to higher education and workforce success, and considerations relevant to their assessment. The goal is to discuss progress on CPS to date and help generate future research on CPS in relation to educational and workforce success.

Genre serves as a useful lens to investigate the range of evidence derived from automated writing evaluation (AWE). To support construct-relevant systems used for writing instruction and assessment, two investigations were conducted that focused on post-secondary writing requirements and faculty perceptions of student writing proficiency. Survey research is described from a national study and a second site study at American University, a 4-year private university in Washington, DC, to illustrate writing requirements and perceptions of writing proficiency in school and workplace settings. A mixed-methods analysis of faculty focus groups in the site study afforded more detailed discussions that were used to understand student writing support needs. Through the lens of genre, study results illustrated how the role of AWE might be expanded and aligned with instruction in four-year post-secondary institutions.

The availability of the Rhetorical Structure Theory (RST) Discourse Treebank has spurred substantial research into discourse analysis of written texts; however, limited research has been conducted to date on RST annotation and parsing of spoken language, in particular, nonnative spontaneous speech. Considering that the measurement of discourse coherence is typically a key metric in human scoring rubrics for assessments of spoken language, we initiated a research effort to obtain RST annotations of a large number of non-native spoken responses from a standardized assessment of academic English proficiency. The resulting inter-annotator κ agreements on the three different levels of Span, Nuclearity, and Relation are 0.848, 0.766, and 0.653, respectively. Furthermore, a set of features was explored to evaluate the discourse structure of non-native spontaneous speech based on these annotations; the highest performing feature showed a correlation of 0.612 with scores of discourse coherence provided by expert human raters.

We present a spoken dialog-based framework for the computerassisted language learning (CALL) of conversational English. In particular, we leveraged the open-source HALEF dialog framework to develop a job interview conversational application. We then used crowdsourcing to collect multiple interactions with the system from non-native English speakers. We analyzed human-rated scores of the recorded dialog data on three different scoring dimensions critical to the delivery of conversational English-fluency, pronunciation and intonation/stress-and further examined the efficacy of automatically-extracted, hand-curated speech features in predicting each of these subscores. Machine learning experiments showed that trained scoring models generally perform at par with the human inter-rater agreement baseline in predicting human-rated scores of conversational proficiency.