A Study of the Effects of Variation of Short‐term Memory Load, Reading Response Length, and Processing Hierarchy on Toefl Listening Comprehension Item Performance

Abstract

This study was conceived in response to criticisms of the current TOEFL listening comprehension test‐item format. Major areas of criticism have included speculation that listening as tested places too much burden on short‐term memory as opposed to comprehension, that a knowledge of reading is required in order to respond successfully, and that many items appear to require mere recall and matching of details rather than higher‐order processing skills. To address these criticisms in turn, a study was designed with 120 ESL learners and three listening tests (comprised of 144 total real and adapted TOEFL test items) to examine the characteristics of item functioning under conditions of stimulus repetition versus nonrepetition, variation of length of aural stimulus passage and of associated numbers of items, shorter versus longer reading response options, and higher versus lower level of processing skills required. Those item types and stimulus conditions that were found to associate with superior item functioning as indicated by estimates of item difficulty, item discriminability, internal consistency reliability, fit to a latent trait model, and convergent and discriminant validity were identified. Results suggested that, while repetition of the stimulus passage predictably tended to reduce item difficulty when control was made for concomitant influences, there was no consistent effect of stimulus passage repetition on item discrimination, Rasch model fit, or discriminant validity across difficulty level. However, there was a tendency for items in the no‐repetition condition to exhibit greater convergent and discriminant validity than items in the one‐repetition condition. Although passage length was confounded with numbers of items per passage and with comprehension hierarchy level, the test with passages of three‐sentence length tended to be more reliable than the test with passages of two‐sentence length, and the test with passages of two‐sentence length tended to be more reliable than the test with passages of one‐sentence length. Also, the test with the longest passages tended predictably to be slightly more difficult than the test with the shortest passages. Item response‐option length was significantly related to item difficulty and Rasch model fit in the direction that items with options that were shortened to about half current TOEFL response‐option length tended to be easier and to exhibit better fit than items with current longer options. Also, items with shortened options showed greater convergent and discriminant validity across levels of difficulty than did items with unshortened options. And, there was a near‐significant tendency for items with shortened options to exhibit better discrimination than items with unshortened options, when concomitant influences were controlled. Comprehension hierarchy level of items, as defined by the length of passage required to respond correctly, was not significantly related to item difficulty except through a complex option‐length‐by‐hierarchy‐ level intera...

What makes listening difficult? Factors affecting second language listening comprehension

“…When L2 listeners are lower proficiency, they may be overwhelmed in trying to process even shorter passages, especially as the "shorter" passages examined by Carrell et al were quite long compared to those examined in earlier studies (~2.5 minutes compared to 10 words in Henning [1990], and < 20 seconds in Kostin [2004] and Nissan et al [1996]). …”

“…Longer passages may be more likely to disrupt comprehension due to overwhelming listeners' working memory storage capacity (Henning, 1990). In addition, the longer a passage is, the more information listeners could miss after encountering information they do not understand.…”

“…Studies that do find a relationship find one that is weak (Henning, 1990) or qualified by interactions (Carrell et al, 2002). However, problems exist in terms of confounding other aspects of the passage with length (Moyer, 2006) or examining passages with a very limited range of lengths (e.g., Nissan et al, 1996).…”

“…However, problems exist in terms of confounding other aspects of the passage with length (Moyer, 2006) or examining passages with a very limited range of lengths (e.g., Nissan et al, 1996). In addition, the range of lengths compared varied greatly across studies (e.g., 2.5 versus 5 minutes in Carrell et al, 2002, and 10 words versus 30 words in Henning, 1990). Several studies have also uncovered other factors more specific than overall passage length…”

“…One rationale behind examining passage length as a factor in L2 listening comprehension is the belief that a greater processing load is introduced by a longer passage (Carroll, 1977, cited in Dunkel, 1991Henning, 1990;Rost, 2006). In service to this concern, the amount of information in a passage may be a more predictive factor for comprehension difficulty than is overall passage length.…”