Exploring Item Characteristics That Are Related to the Difficulty of Toefl Dialogue Items

Abstract

The purpose of this study is to explore the relationship between a set of item characteristics and the difficulty of TOEFL ® dialogue items. Identifying characteristics that are related to item difficulty has the potential to improve the efficiency of the item-writing process The study employed 365 TOEFL dialogue items, which were coded on 49 variables, including 5 significant variables reported in Nissan, DeVincenzi, and Tang (1996). Of the 5 significant variables in Nissan et al., 3 correlated significantly with item difficulty in this study. Another 11 met a critical probability criterion. These 11 included representatives from three broad categories of variables: 2 in the category of word-level factors, 1 in the category of discourse-level factors, and 8 in the category of task-processing factors. Multiple regression analyses indicate that the variables in this study account for about 40% of the variance in item difficulty. ETS administers the TOEFL program under the general direction of a policy board that was established by, and is affiliated with, the sponsoring organizations. Members of the TOEFL Board (previously the Policy Council) represent the College Board, the GRE Board, and such institutions and agencies as graduate schools of business, junior and community colleges, nonprofit educational exchange agencies, and agencies of the United States government.A continuing program of research related to the TOEFL test is carried out in consultation with the TOEFL Committee of Examiners. Its 12 members include representatives of the TOEFL Board and distinguished English as a second language specialists from the academic community. The Committee advises the TOEFL program about research needs and, through the research subcommittee, reviews and approves proposals for funding and reports for publication. Members of the Committee of Examiners serve four-year terms at the invitation of the Board; the chair of the committee serves on the Board.Because the studies are specific to the TOEFL test and the testing program, most of the actual research is conducted by ETS staff rather than by outside researchers. Many projects require the cooperation of other institutions, however, particularly those with programs in the teaching of English as a foreign or second language and applied linguistics. Representatives of such programs who are interested in participating in or conducting TOEFL-related research are invited to contact the TOEFL program office. All TOEFL research projects must undergo appropriate ETS review to ascertain that data confidentiality will be protected. Current (2003Current ( -2004 members of the TOEFL Committee of Examiners are:

What makes listening difficult? Factors affecting second language listening comprehension

“…When L2 listeners are lower proficiency, they may be overwhelmed in trying to process even shorter passages, especially as the "shorter" passages examined by Carrell et al were quite long compared to those examined in earlier studies (~2.5 minutes compared to 10 words in Henning [1990], and < 20 seconds in Kostin [2004] and Nissan et al [1996]). …”

“…However, Thompson and Rubin found that passage length interacted with other qualities of the passage: longer dramatic segments were better tolerated than longer news reports, suggesting that number of speakers, structure of the passage, or topic may interact with length. Nissan, DeVincenzi, and Tang (1996) found no effect of overall passage length in seconds on listening comprehension item difficulty, and Kostin (2004) found no significant effect of the total number of words in the passage on comprehension difficulty. However, the passages examined in both these studies were fairly short (< 20 seconds in length).…”

“…1 Harrington and Sawyer (1992) 2 Vandergrift et al (2006) 3 Tsui and Fullilove (1998) 4 Field (2004) 5 Goh (2000) 6 Tyler (2001) 7 Nation (2001) 8 e.g., Rupp, Garcia, & Jamieson (2001) 9 e.g., Nissan, DeVincenzi, & Tang (1996) 10 Moyer (2006) 11 Rupp et al (2001) 12 Rupp et al (2001) 13 Chang & Dunkel (1992) 14 Chaudron (1983) 15 Nissan et al (1996); Kostin (2004) 16 Kostin (2004) 17 e.g., Pica, Young, & Doughty (1987) 18 Taguchi (2008) 19 Taguchi (2005) 20 Nissan et al (1996) 21 Freedle & Kostin (1992 22 Sasaki (2000) 23 Kostin (2004) 24 Tannen (1982) 25 e.g., Kiany & Jalali (2006) 26 Odlin (1989) 27 Freedle & Kostin (1992) 28 Ying-hui (2006) 29 e.g., Jung (2003) 30 Chaudron & Richards (1986) 31 Freedle & Kostin (1996 32 Major et al (200...…”

“…The studies that have investigated the impact of the syntactic structure of a passage have done so by considering the degree of subordination (Blau, 1990;Cervantes & Gainer, 1992;Pica et al, 1987), the number of negatives (Kostin, 2004;Nissan et al, 1996;Yanagawa & Green, 2008;Ying-hui, 2006), the number of dependent clauses (Kostin, 2004;Ying-hui, 2006), or the number of references (Kostin, 2004;Ying-hui, 2006) in the passage.…”

“…55 Other factors that impact difficulty (e.g., speech rate) Rupp et al (2001) 12 Rupp et al (2001) 13 Chang & Dunkel (1992) 14 Chaudron (1983) 15 Nissan et al (1996); Kostin (2004) 16 Kostin (2004) 17 e.g., Pica, Young, & Doughty (1987) 18 Taguchi (2008) 19 Taguchi (2005) 20 Nissan et al (1996) 21 Freedle & Kostin (1992 22 Sasaki (2000) 23 Kostin (2004) 24 Tannen (1982) 25 e.g., Kiany & Jalali (2006) 26 Odlin (1989) 27 Freedle & Kostin (1992) 28 Ying-hui (2006) 29 e.g., Jung (2003) 30 Chaudron & Richards (1986) 31 Freedle & Kostin (1996 …”