Writing Quality Predictive Modeling: Integrating Register-Related Factors

Abstract

The primary purpose of this study is to investigate the degree to which register knowledge, register-specific motivation, and diverse linguistic features are predictive of human judgment of writing quality in three registers—narrative, informative, and opinion. The secondary purpose is to compare the evaluation metrics of register-partitioned automated writing evaluation models in three conditions: (1) register-related factors alone, (2) linguistic features alone, and (3) the combination of these two. A total of 1006 essays ( n = 327, 342, and 337 for informative, narrative, and opinion, respectively) written by 92 fourth- and fifth-graders were examined. A series of hierarchical linear regression analyses controlling for the effects of demographics were conducted to select the most useful features to capture text quality, scored by humans, in the three registers. These features were in turn entered into automated writing evaluation predictive models with tuning of the parameters in a tenfold cross-validation procedure. The average validity coefficients (i.e., quadratic-weighed kappa, Pearson correlation r, standardized mean score difference, score deviation analysis) were computed. The results demonstrate that (1) diverse feature sets are utilized to predict quality in the three registers, and (2) the combination of register-related factors and linguistic features increases the accuracy and validity of all human and automated scoring models, especially for the registers of informative and opinion writing. The findings from this study suggest that students’ register knowledge and register-specific motivation add additional predictive information when evaluating writing quality across registers beyond that afforded by linguistic features of the paper itself, whether using human scoring or automated evaluation. These findings have practical implications for educational practitioners and scholars in that they can help strengthen consideration of register-specific writing skills and cognitive and motivational forces that are essential components of effective writing instruction and assessment.

Extending a Pretrained Language Model (BERT) using an Ontological Perspective to Classify Students’ Scientific Expertise Level from Written Responses

“…It aims to capture general features of answer quality (e.g., organization, style, and persuasiveness) relying on raters' sensitivities to the construct (Klein et al, 1998), backgrounds and knowledge (Zhai, Haudek, Stuhlsatz, and Wilson, 2020). It is widely documented that the reliability of holistic scores can be in uenced by various sources of measurement errors such as raters' effects, the writer's individual characteristics, and the writing prompt itself used to elicit a writing sample (Barkaoui, 2007;Wang and Troia, 2023).…”

“…Given the limitations in automated scoring models, it is vital to consider integrating more nuanced writing constructs in CR assessments. It may include individual differences related to sociocultural and cognitive factors (e.g., Crossley, Allen, Snow, and McNamara, 2016), along with academic attributes (e.g., Murphy and Yancey, 2008;Wang and Troia, 2023) such as keywords, words frequency, sentence structure, text length, given their substantial in uence on essay quality and characteristics. For instance, coherent and quali ed CRs tend to demonstrate a greater and more appropriate use of academic, sophisticated vocabulary, coupled with a more advanced level of syntactic complexity in explaining scienti c phenomena (Wang et al, 2023).…”

“…It may include individual differences related to sociocultural and cognitive factors (e.g., Crossley, Allen, Snow, and McNamara, 2016), along with academic attributes (e.g., Murphy and Yancey, 2008;Wang and Troia, 2023) such as keywords, words frequency, sentence structure, text length, given their substantial in uence on essay quality and characteristics. For instance, coherent and quali ed CRs tend to demonstrate a greater and more appropriate use of academic, sophisticated vocabulary, coupled with a more advanced level of syntactic complexity in explaining scienti c phenomena (Wang et al, 2023). An exemplary approach in this domain is the Constructed Response Classi er (CRC; see Noyes et al, 2020) which comprises an 8-classi cation ML-based algorithm ensemble implemented in R program (Jurka et al, 2012).…”