Criterion‐related Validity of the Toefl Ibt Listening Section

Abstract

The study investigated the criterion-related validity of the Test of English as a ForeignLanguage™ Internet-based test (TOEFL ® iBT) Listening section by examining its relationship to a criterion measure designed to reflect language-use tasks that university students encounter in everyday academic life: listening to academic lectures. The design of the criterion measure was informed by students' responses to a survey on the frequency and importance of various classroom tasks that require academic listening, and the relationship of these tasks to successful course completion. The criterion measure consisted of three videotaped lectures (in physics, history, and psychology) and included tasks created by content experts who are former university professors of the relevant content area. These tasks reflected what the content experts expected students to have comprehended during the lecture.The criterion measure and the TOEFL iBT Listening section were administered to nonnative speakers of English who were enrolled in undergraduate and graduate programs. Data from 221 participants were analyzed. Substantial correlations were observed between the criterion measure and the TOEFL iBT Listening section score for the entire sample and for subgroups (Pearson correlation coefficients ranging from .56 to .74 and disattenuated correlations ranging from .62 to .82). Moreover, the analysis of the mean scores on the criterion measure for different ability groups indicated that participants who scored at or above typical cut scores for international student admission to academic programs (i.e., TOEFL iBT Listening section score of 14 or above) scored, on average, nearly 50% or more on the criterion measure, demonstrating reasonable comprehension of the academic lectures. ETS administers the TOEFL program under the general direction of a policy board that was established by, and is affiliated with, the sponsoring organizations. Members of the TOEFL Board (previously the Policy Council) represent the College Board, the GRE Board, and such institutions and agencies as graduate schools of business, two-year colleges, and nonprofit educational exchange agencies.Since its inception in 1963, the TOEFL has evolved from a paper-based test to a computer-based test and, in 2005, to an Internet-based test, TOEFL iBT. One constant throughout this evolution has been a continuing program of research related to the TOEFL test. From 1977 to 2005, nearly 100 research and technical reports on the early versions of TOEFL were published. In 1997, a monograph series that laid the groundwork for the development of TOEFL iBT was launched. With the release of TOEFL iBT, a TOEFL iBT report series has been introduced.Currently this research is carried out in consultation with the TOEFL Committee of Examiners. Its members include representatives of the TOEFL Board and distinguished English as a second language specialists from the academic community. The Committee advises the TOEFL program about research needs and, through the research subcommittee...

Investigating the Relationship Between Test Preparation and TOEFL iBT® Performance

“…A third limitation is that no item‐level data were available for this study, and therefore we were not able to provide the reliability and intercorrelations of the four skill areas. However, readers are referred to several TOEFL iBT validity studies for such information (e.g., Attali, , on writing; Liu, , on reading; Sawaki & Nissan, , on listening; Sawaki, Stricker, & Oranje, , on all four skills). According to the Sawaki et al () study, the correlations among the reading, listening, and writing sections ranged from .86 to .89.…”

Improvement of Japanese adults' English speaking skills via experiences speaking to a robot

“…Audiobooks are ideally recorded at a rate of 150–160 words per minute, which is the range that people can comfortably hear and vocalize words (Williams, ). The speech rate of TOEIC iBT tests is said to be in the range of 110–200 words per minute (Sawaki & Nissan, ). The average speaking speed of the pretest was 55.2 words per minute, which is quite slow.…”

A Study of the Use of the TOEFL iBT® Test Speaking and Listening Scores for International Teaching Assistant Screening

“…Sawaki and Nissan () conducted a criterion‐related validity study of the listening section of the TOEFL iBT. In their study, Sawaki and Nissan surveyed 145 undergraduate and graduate students at four North American universities, asking them about the types of listening tasks that were required in their university classes and course assignments.…”

“…As stated earlier, a number of studies (e.g., Biber & Gray, ; Bridgeman et al, ) have been conducted providing empirical evidence for the validity argument in support of the TOEFL iBT Speaking. However, there does not seem to be the same amount of research analyzing the TLU domain of academic listening as part of the validity argument of the TOEFL iBT Listening section (other than Sawaki & Nissan, ). Indeed, Wagner (, , ) reviewed large‐scale English proficiency tests used for North American university admissions (including the TOEFL iBT), and Wagner and Wagner () did the same with large‐scale English proficiency tests used in Asia.…”

“…They noted that the spoken texts used on the listening sections of the tests lack many of the characteristics of real‐world spoken language, including connected speech, hesitation phenomena, spoken grammatical norms, and the nonverbal information transmitted by the speaker. In addition, while these studies (Biber & Gray, ; Bridgeman et al, ; Sawaki & Nissan, ) have provided validity evidence for the TOEFL iBT Speaking and Listening sections, they have focused on using the TOEFL iBT as a measure of test takers' ability to perform as a student at a North American university. But the use of the TOEFL iBT as a screening measure for instructors would seem to be (at least in part) targeting a different language use domain (Farnsworth, ; Farnsworth & Wagner, ).…”