Validating the Revised Test of Spoken English Against a Criterion of Communicative Success

Abstract

A conununicative competence orientation wasadopted to study the validity of test score inferences derived from the revised Test of Spoken English (fSE). To implement the approach, a sample of undergraduate students, primarily native speakers of English, provided a variety of reactions to, andjudgments of, the test responses ofa sample ofTSE examinees. The TSE scores ofthese examinees, previously determined by official TSE raters, spanned the full range ofTSE score levels. Undergraduate students were selected as "evaluators" because they, more than most other groups, are likely to interact with TSE examinees, many of whom become teaching assistants.Student evaluations were captured by devising and administering a secondary listening test (SLT) to assess students' understanding of TSE examinees' speech, as represented by their taped responses to tasks on the Test of Spoken English. The objective was to determine the degree to which official TSE scores are predictive of listeners' ability to understand the messages conveyed by TSE examinees. Analyses revealed a strong association between TSE score levels and the judgments, reactions, and understanding of listeners. This finding applied to all TSE tasks and to nearly all of the several different kinds of evaluations made by listeners. Along with other information, the evidence gathered here should help the TSE program meet professional standards for test validation. The procedures may also prove useful in future test development efforts as a way of determining the difficulty of speaking tasks (and possibly writing tasks).The Test ofEnglish as a Foreign Language (TOEFLIll) was developed in 1963 by the National Council on the Testing of English as a Foreign Language. The Council was formed through the cooperative effort of more than 30 public and private organizationsconcerned with testing the English proficiency of nonnative speakers of the language applying for admission to institutions in the United States. In 1965,Educational Testing Service (ETSIIl) and the College Board assumed joint responsibility for the program. In 1973,a cooperative arrangement for the operationof the program was entered into by ETS, the College Board, and the Graduate Record Examinations (GRE~) Board. The membership of the College Board is composed of schools, colleges, school systems, and educational associations; GRE Board members are associated with graduate education.ETS administers the TOEFL program under the general direction of a Policy Council that was established by, and is affiliated with, the sponsoring organizations. Members of the Policy Council represent the College Board, the GRE Board, and such institutions and agencies as graduate schools of business,junior and community colleges, nonprofit educational exchange agencies, and agencies of the United States government.A continuing program of research related to the TOEFL test is carried out under the direction of the TOEFL Research Committee. Its six members include representatives of the Policy Council, the TOEFL Comm...

Mitigating U.S. Undergraduates’ Attitudes Toward International Teaching Assistants

“…Building on the research of Rubin () and others (e.g., Powers, Schedl, Wilson‐Leung, & Butler, ), a questionnaire was developed to obtain background information regarding participants’ experiences with ITA courses. In addition to demographic questions, the background questionnaire included raters’ experiences with and general attitudes about nonnative speakers and their reactions to the courses taught by ITAs.…”

A Study of the Use of the TOEFL iBT® Test Speaking and Listening Scores for International Teaching Assistant Screening

“…The SPEAK test is composed of retired versions of the Test of Spoken English (TSE). Powers, Schedl, Wilson Leung, and Butler () performed a validation study of the TSE in which undergraduate students (predominantly L1 English speakers) rated the spoken texts of TSE test takers. In addition, a listening comprehension test was composed using the spoken responses of the TSE test takers.…”

A Historical Survey of Research Regarding Constructed-Response Formats

“…Envisioning TOEFL from that perspective was a likely outcome of the conference (Duran et al 1987). Evidence of the acceptance of the communicative competence construct can be seen in its use to validate TSE scores (Powers et al 1999b), and in the framework for incorporating a speaking component in a revised TOEFL (Butler et al 2000). The first step in the development of an operational computer-based speaking test was the TOEFL Academic Speaking Test (TAST), a computer-based test intended to familiarize TOEFL test takers with the new format.…”