Validating the revised Test of Spoken English against a criterion of communicative success

Abstract

A communicative competence orientation was taken to study the validity of test-score inferences derived from the revised Test of Spoken English (TSE). To implement the approach, a sample of undergraduate students, primarily native speakers of English, provided a variety of reactions to, and judgements of, the test responses of a sample of TSE examinees. The TSE scores of these examinees, previously determined by official TSE raters, spanned the full range of TSE score levels. Undergraduate students were selected as ‘evaluators’ because they, more than most other groups, are likely to interact with TSE examinees, many of whom become teaching assistants. Student evaluations were captured by devising and administering a secondary listening test (SLT) to assess students’ understanding of TSE examinees’ speech, as represented by their taped responses to tasks on the TSE. The objective was to determine the degree to which official TSE scores are predictive of listeners’ ability to understand the messages conveyed by TSE examinees. Analyses revealed a strong association between TSE score levels and the judgements, reactions and understanding of listeners. This finding applied to all TSE tasks and to nearly all of the several different kinds of evaluations made by listeners. Along with other information, the evidence gathered here should help the TSE program meet professional standards for test validation. The procedures may also prove useful in future test-development efforts as a way of determining the difficulty of speaking tasks (and possibly writing tasks).

Mitigating U.S. Undergraduates’ Attitudes Toward International Teaching Assistants

“…Background questionnaire. Building on the research of Rubin (1992) and others (e.g., Powers, Schedl, Wilson-Leung, & Butler, 1999), a questionnaire was developed to obtain background information regarding participants' experiences with ITA courses. In addition to demographic questions, the background questionnaire included raters' experiences with and general attitudes about nonnative speakers and their reactions to the courses taught by ITAs.…”

Predicting communicative effectiveness in the international workplace: Support for TOEIC® Speaking test scores from linguistic laypersons

“…Studies of oral language proficiency have alternately described this population as naive listeners (Schmidgall, 2013), evaluators (Powers et al, 1999), and ordinary listeners (Bridgeman et al, 2012). Although researchers (e.g., Elder et al, 2017) have invoked the notion of linguistic laypersons more narrowly to focus on content-area specialists who are not language specialists, for the purpose of this study we adopt the broader definition of this population as non-language experts who are actively engaged in the TLU domain.…”

“…Linguistic laypersons' lack of training, together with the possibility that training may actually reduce the generalizability of their evaluations, underscores the appropriateness of utilizing intuitive, high-inference measures to capture layperson evaluations. In efforts to capture impressionistic, non-expert judgments of L2 communication ability, researchers in applied linguistics have proposed a number of related constructs, including communicative effectiveness (Björkman, 2011;Hu, 2017;Sato, 2012), communicative ability (Sato, 2014), communicative adequacy (Pallotti, 2009;Révész et al, 2016), communicative understanding (Bridgeman et al, 2012), communicative success (Powers et al, 1999), and functional adequacy (de Jong et al, 2012;Kuiken & Vedder, 2017, 2018.…”