Using observation checklists to validate speaking-test tasks

Abstract

Test-task validation has been an important strand in recent revision projects for University of Cambridge Local Examinations Syndicate (UCLES) examinations. This article addresses the relatively neglected area of validating the match between intended and actual test-taker language with respect to a blueprint of language functions representing the construct of spoken language ability. An observation checklist designed for both a priori and a posteriori analysis of speaking task output has been developed. This checklist enables language samples elicited by the task to be scanned for these functions in real time, without resorting to the laborious and somewhat limited analysis of transcripts. The process and results of its development, implications and further applications are discussed.

A teacher-verification study of speaking and writing prototype tasks for a new TOEFL

“…For tests of language abilities, many theorists have recently argued that assessment tasks need to "somehow capture or recreate…the essence of language use" (Bachman, 1990, p. 300), urging that authenticity should be a fundamental criterion for the validity of tasks that aim to evaluate how well people can really communicate in a language (Bachman & Palmer, 1996;Lewkowicz, 2000;Spence-Brown, 2001;Spolsky, 1985). In turn, to achieve educational relevance, test tasks should elicit performance that is congruent with instructors' perceptions of their students' abilities, as numerous different inquiries in secondlanguage assessment have recently emphasized and demonstrated (e.g., Brindley, 1998Brindley, , 2000Chalhoub-Deville, 1995;Elder, 1993;Epp & Stawychny, 2001;Grant, 1997;North, 1995North, , 2000Stansfield & Kenyon, 1996;Sullivan, Weir, & Saville, 2002; Teachers of English to Speakers of Other Languages, 1998). Nonetheless, surprisingly few studies (apart from those cited in the preceding sentence) have systematically sought the input of experienced teachers for the design or validation of second-language tests, despite a published literature that has established the reliability and informative value of teachers' judgments of their students' abilities or achievements in relation to standardized tests in various domains of education (e.g., Griffin, 1995;Hoge & Coladarci, 1989;Miesels, Bickel, Nicholson, Xue, & Atkins-Burnett, 2001;Sharpley & Edgar, 1986).…”

Looking beyond scores: validating a CEFR-based university speaking assessment in Mainland China

“…The useful tripartite distinction of functional resources between informational, interactional and interaction management functions were remained (O'Sullivan et al 2002). The checklist was revised after a preliminary coding drawing upon the established checklist with a small proportion of data.…”

Validation of the Oral Interaction Strategy Scale for speakers of Chinese as the first language in elementary schools

“…Studies have shown that clarification plays an important role in the negotiation of meaning during L2 conversation (Benson, Fischer, Geluso, & Von Joo, 2013;Nakahama, Tyler & Lier, 2001;Nakatani, 2010;Pica, 1994). In L1, it is quite often that, in order to make one's own views more convincible, one will give explanations, examples or reasons, but from a non-linguistic perspective (Zhu, 2014b;O'Sullivan, Weir & Saville, 2002).…”

“…In L2 research, it corresponds to responding (to a content-related question asked by a member of the group) in Bejarano et al (1997). During L1 oral communication, expressing one's agreement or disagreement on a certain view is a common behavior; it is also a manifestation of students' ability in higher level thinking such as challenging and critical thinking (Zhu, 2016a;O'Sullivan, Weir & Saville, 2002). iv) Correcting errors (e.g., item 13, I correct others when I notice a mistake (either ideas or language).…”