Probing the relative importance of different attributes in L2 reading and listening comprehension items: An application of cognitive diagnostic models

Abstract

The present study examines the relative importance of attributes within and across items by applying four cognitive diagnostic assessment models. The current study utilizes the function of the models that can indicate inter-attribute relationships that reflect the response behaviors of examinees to analyze scored test-taker responses to four forms of TOEFL reading and listening comprehension sections. The results are interpreted to determine whether the subskills defined in each subtest contribute equally within an item and across items. The study also discusses whether the empirical results support the claims for compensatory processing of L2 comprehension skills and presents practical implications of the findings. The article concludes with the limitations of the study and suggestions for future research. KeywordsAttributes in second language comprehension tests, cognitive diagnostic models, relative difficulty of L2 subskills, relative importance/contribution of L2 reading or listening skills Research on second language attributes that are assessed through reading and listening comprehension test items has been conducted using a variety of methods. First, to identify the specific subskills measured by L2 reading and listening tests, many studies have employed introspective or retrospective methods such as verbal protocols or interviews Downloaded from 2 Language Testing to elicit learners' reports. A number of studies have also employed statistical and/or psychometrical analysis techniques that focused on surface task characteristics such as item difficulty (Sawaki, Kim, & Gentile, 2009).In relation to this line of studies, another body of research has focused on the hierarchical relationships among L2 receptive subskills in terms of difficulty. These studies have examined whether the existing or suspected hierarchy of attributes could be empirically validated by comparing the hierarchy with actual student performance (e.g., Alderson & Lukmani, 1989) and by investigating the performance differences on the subskills among various levels of learners (Mecartty, 1998).Generalizing these findings to other L2 comprehension tests faces the limitation that most of the test items in these studies were matched to only one subskill. Thus, the relative roles of the attributes are not well known when items are coded for more than one subskill. Considering that many L2 test items are likely to require more than one skill, it seems necessary to scrutinize the relationships among them by analyzing tests at the item level.The present study attempts to fill this research gap by applying psychometric models of cognitive diagnostic assessment to probe the processes of multiple (in this study, two) attributes within an item or across items. As quantitative analysis tools of a measurement procedure, cognitive diagnostic models have dual functions. They primarily estimate the skill mastery profiles of test-takers by classifying students according to their mastered skills and extract detailed information about their strengths and wea...

Assessing Second Language Listening Over the Past Twenty Years: A Review Within the Socio-Cognitive Framework

“…A popular approach to investigating listening subskills is the use of cognitive diagnosis models. Listening subskills were found to be different in terms of various grain sizes ( Sawaki et al, 2009 ), and the contribution of a particular listening subskill was not consistent across items ( Yi, 2017 ), indicating the vague definition of L2 listening subskills ( Aryadoust, 2020 ). To address this gap, Aryadoust (2020) used the document co-citation analysis to give a systematic review of research on comprehension subskills.…”

An Optimal Choice of Cognitive Diagnostic Model for Second Language Listening Comprehension Test

“…Weir (1993) is along a similar line of defining important listening subskills only. The idea of major listening subskills benefits listening comprehension test development and studies in terms of the construct validity of these tests, especially studies with CDA (Buck and Tatsuoka, 1998;Lee and Sawaki, 2009a;Sawaki et al, 2009;Yi, 2017).…”

“…In addition, Lee and Sawaki (2009a) concluded that compensatory and non-compensatory models produced strikingly comparable diagnostic results. Yi (2017), however, argued that a compensatory model (C-RUM) was the best to interpret the listening subskill relationships. As both compensatory and non-compensatory CDMs were applied to L2 listening comprehension tests in the literature, CDM selection for L2 listening comprehension tests is still inconclusive and deserves further exploration.…”

“…Previous studies, though sparse, have shown the feasibility of applying CDMs to L2 listening comprehension tests and demonstrated the potential usefulness of cognitive diagnostic approaches (CDAs) to understanding various subskills (Buck and Tatsuoka, 1998;Lee and Sawaki, 2009a;Sawaki et al, 2009;Aryadoust, 2011;Meng, 2013;Yi, 2017). However, it is not very clear which CDM should be used for L2 listening comprehension tests.…”

“…Wrong CDM(s) for a given dataset will lead to the wrong classification of examinees and misleading feedback. In addition, CDMs can provide information on the underlying inter-attribute relationships (Yi, 2017), that is, whether a compensatory or noncompensatory inter-attribute relationship can produce correct answers. Wrong CDM(s) for given data will generate a wrong interpretation of inter-attribute relationships and skill mastery status.…”