Domain Description: Validating the Interpretation of the TOEFL iBT® Speaking Scores for International Teaching Assistant Screening and Certification Purposes

Abstract

In the past 2 decades, there has been an increasing tendency to use scores from the TOEFL iBT® Speaking test for decisions regarding the certification of international graduate students as teaching assistants at North American universities. To obtain validity evidence in support of the usefulness of the speaking scores for this secondary use of the test, this study adopted the argument‐based approach to validation. Focusing on the domain description inference in the TOEFL interpretive argument, the study investigated whether the language functions elicited by TOEFL iBT Speaking tasks can be identified in authentic discourse produced by international teaching assistants (ITAs) with different instructional roles. We compiled and analyzed 2 corpora—a TOEFL iBT speech corpus and an ITA speech corpus. The TOEFL corpus contained 2,738 responses to integrated and independent tasks. The ITA corpus of 119 spoken texts included multiple disciplines and 3 instructional genres: lab, recitation, and lecture. The 2 corpora were manually annotated using the knowledge framework (Mohan, 1986), which is a heuristic in systemic functional linguistics (SFL) used to identify knowledge structures (KSs) and language functions based on how linguistic choices function in the discourse. Then, the following types of data were quantitatively analyzed: discourse units annotated per KS category, discourse units annotated per language function, KS categories occurring in each text, language functions occurring in each text, and KSs and functions from each component of the spoken corpora. The corpus data revealed how the language functions were realized and how they varied. Overall, the results indicated that TOEFL iBT Speaking tasks elicit most of the language functions identified in ITA discourse, suggesting that this test accounts for the functional language ability necessary for effective instructional performance as a teaching assistant in the target domain of language use. The discrepancy detected in the use of some functions pertaining to 2 KSs warrants further examination of the extent to which it may impact secondary test use and score interpretation.

Questioning in TOEFL iBT speaking test: a case of washback and construct underrepresentation

“…The TOEFL iBt speaking test has been subject to ongoing validation studies. Some studies (e.g., Bridgeman et al, 2012;Cotos & Chung, 2018;Cumming et al, 2005;Enright & Tyson, 2008;Ockey et al, 2015) have examined the alignment between the test and TLUS tasks and reported TOEFL iBT to be a valid measure of communicative ability owing to setting up an actual situation requiring applicants to demonstrate their capacity to perform various TLUS tasks in the test. Cotos and Chung (2018) sought to explore to what extent the language functions elicited by TOEFL iBT speaking tasks are in line with those fulfilled by international teaching assistants in the authentic discourse.…”

“…Some studies (e.g., Bridgeman et al, 2012;Cotos & Chung, 2018;Cumming et al, 2005;Enright & Tyson, 2008;Ockey et al, 2015) have examined the alignment between the test and TLUS tasks and reported TOEFL iBT to be a valid measure of communicative ability owing to setting up an actual situation requiring applicants to demonstrate their capacity to perform various TLUS tasks in the test. Cotos and Chung (2018) sought to explore to what extent the language functions elicited by TOEFL iBT speaking tasks are in line with those fulfilled by international teaching assistants in the authentic discourse. They concluded that "TOEFL iBT speaking tasks elicit most of the language functions identified in the discourse, suggesting that this test accounts for the functional language ability necessary for effective instructional performance as a teaching assistant in the target domain of language use" (p.1).…”