Designing Academic Writing Analytics for Civil Law Student Self-Assessment

Abstract

Research into the teaching and assessment of student writing shows that many students find academic writing a challenge to learn, with legal writing no exception. Improving the availability and quality of timely formative feedback is an important aim. However, the time-consuming nature of assessing writing makes it impractical for instructors to provide rapid, detailed feedback on hundreds of draft texts which might be improved prior to submission. This paper describes the design of a natural language processing (NLP) tool to provide such support. We report progress in the development of a web application called AWA (Academic Writing Analytics), which has been piloted in a Civil Law degree. We describe: the underlying NLP platform and the participatory design process through which the law academic and analytics team tested and refined an existing rhetorical parser for the discipline; the user interface design and evaluation process; and feedback from students, which was broadly positive, but also identifies important issues to address. We discuss how our approach is positioned in relation to concerns regarding automated essay grading, and ways in which AWA might provide more actionable feedback to students. We conclude by considering how this design process addresses the challenge of making explicit to

Are You Being Rhetorical? A Description of Rhetorical Move Annotation Tools and Open Corpus of Sample Machine-Annotated Rhetorical Moves

“…the "concept-matching" framework (Sándor, Kaplan, & Rondeau, 2006)). Thus, for example, as outlined in Knight, Buckingham Shum, Ryan, Sándor, and Wang (2018), "contrast" sentences include syntactically related words that instantiate the concept of both "idea" and "contrast. "…”

Untitled

“…Law undergraduates, like students of any other discipline, are expected to have already acquired certain basic proficiency in writing in terms of expression and lower order concerns (LOCs) when they commence university education so that they can have a smooth transition into the world of academic writing -argumentative essays, project reports, and research papers among other forms of writing -across disciplines and contexts (Cumming, 1989;Griggs, 1996;Knight et al, 2018;Paltridge, 2018;Wingate et al, 2011 ). Writing instructors at law school expect students to understand what "specificity", "precision" and "concision" in academic writing are and also to understand the distinction between relevant and irrelevant knowledge in a context and to supply discretionary information in an academic writing context (Bacha & Bahous 2008;Horowitz 1986;Street, 2004;Wilkes et al, 2015).…”

Evaluating Machine Learning Approaches to Classify Pharmacy Students’ Reflective Statements

“…The extracted features were inspired by existing work in reflective writing [3,15]. Linguistic Inquiry and Word Count (LIWC) is a linguistic analysis product [16] which extracts approximately 90 linguistic measures indicative of a large set of psychological processes (e.g., affect, cognition, biological process, drives), personal concerns (e.g., work, home, leisure activities) and linguistic categories (e.g., nouns, verbs, adjectives is an open source software platform (heta.io) focusing on providing actionable feedback to support academic writing, such as analytical writing [11] and reflective writing [3]. Gibson et al [3] used the concept-matching rhetorical analysis framework [21] to automatically detect sentences indicating three key reflective rhetorical moves: Context (initial thoughts and feelings about a significant experience), Challenge (the challenge of new surprising or unfamiliar ideas, problems or learning experience) and Change (potential solution and learning opportunities).…”

“…Natural language processing could assist if student texts can be analyzed, automatically coded (classified) according to a scheme, and in a learning context, helpful feedback given. However, there is very little work in this field to date, with research, and products, in Automated Writing Evaluation (AWE) dominated by more common genres of writing such as persuasive essays, literature review or research proposals [8][9][10][11]. The work in reflective writing to date uses either a rule-based [12] or machine learning approach to classify reflective sentences [13], reflective passages [14], forum posts [15], with only one example of automated feedback deployed with students [3].…”