An Optimal Choice of Cognitive Diagnostic Model for Second Language Listening Comprehension Test

Abstract

Cognitive diagnostic models (CDMs) show great promise in language assessment for providing rich diagnostic information. The lack of a full understanding of second language (L2) listening subskills made model selection difficult. In search of optimal CDM(s) that could provide a better understanding of L2 listening subskills and facilitate accurate classification, this study carried a two-layer model selection. At the test level, A-CDM, LLM, and R-RUM had an acceptable and comparable model fit, suggesting mixed inter-attribute relationships of L2 listening subskills. At the item level, Mixed-CDMs were selected and confirmed the existence of mixed relationships. Mixed-CDMs had better model and person fit than G-DNIA. In addition to statistical approaches, the content analysis provided theoretical evidence to confirm and amend the item-level CDMs. It was found that semantic completeness pertaining to the attributes and item features may influence the attribute relationships. Inexplicable attribute conflicts could be a signal of suboptimal model choice. Sample size and the number of multi-attribute items should be taken into account in L2 listening cognitive diagnostic modeling studies. This study provides useful insights into the model selection and the underlying cognitive process for L2 listening tests.

Exploring the Role of English as a Foreign Language Receptive Skills and Learning Strategy Usage in the Ability to Acquire and Apply Knowledge at the Beginning of Higher Education

“…It also requires a number of skills to understand what others are saying. These involve recognition of sound articulated by the speaker, perception of intonation patterns, construal of the essence of what is being said, understanding the vocabulary and grammar patterns, and placing the information into context ( Kothacheruvu, 2014 ; Yuzar and Rejeki, 2020 ; Dong et al, 2021 ).…”

Modeling mediation in the dynamic assessment of listening ability from the cognitive diagnostic perspective

“…Stage 1: Diagnostic test and Q-matrix development Items for the three diagnostic listening tests were selected from the two item pools. IP1 items were selected from previous cognitive diagnostic studies (Dong et al, 2021;Ma & Meng, 2014;Meng, 2013). IP2 items were retrofitted and adapted from a placement test for diagnostic purposes.…”

“…Considering the complex nature of the L2 listening process and the advantage of CDA in dealing with multiple attributes, CDA‐based L2 listening assessment research has appeared in the last three decades (Aryadoust, 2021; Dong et al., 2021; Meng, 2013; Sawaki et al., 2009; Yi, 2017). This research applied different CDMs to analyze the listening section of large‐scale tests, yielding an array of listening attributes (e.g., understanding words, understanding syntactic structure, inferencing, extracting facts and details, summarizing main ideas), which were proven critical for comprehension of listening materials.…”

Assessing Students’ Translation Competence: Integrating China’s Standards of English With Cognitive Diagnostic Assessment Approaches

“…Taking a step forward, quite a few studies have explored the extent to which diagnostic results could help discover the relationships among attributes (e.g., Chen and Chen, 2016a ; Ravand and Robitzsch, 2018 ; Du and Ma, 2021 ) and could differ across different proficiency groups (e.g., Kim, 2010 , 2011 ; Fan and Yan, 2020 ). Among all those studies, a notable phenomenon is that assessment of receptive language skills such as reading and listening (e.g., von Davier, 2008 ; Jang, 2009b ; Lee and Sawaki, 2009b ; Kim, 2015 ; Chen and Chen, 2016b ; Yi, 2017 ; Aryadoust, 2021 ; Dong et al, 2021 ; Toprak and Cakir, 2021 ; Min and He, 2022 ) gained much more attention than that of productive ones such as writing ( Kim, 2010 , 2011 ; Xie, 2017 ; Effatpanah et al, 2019 ; He et al, 2021 ). “One possible reason is that different test methods are used to assess these two types of skills” ( He et al, 2021 , p. 1).…”