This review paper constitutes the first step within a larger research effort to develop an interactive pragmatics learning tool for second and foreign language (L2) learners and users of English. The tool will primarily endeavor to support pragmatics learning within the language use domain “workplace.” Given this superordinate objective, this paper is subdivided into 2 parts. In the first section, we provide a detailed overview of previous (empirical) research, theories, and frameworks of communicative competence to review the role of pragmatics as an essential component of L2 communicative language ability. A principled, systematic, and exhaustive literature search was conducted via key word searches, and the selected literature was categorized and coded using NVivo 10 software. Next, 12 distinct models of communicative language ability that contain components of pragmatic knowledge were identified and analyzed. The commonly identified constitutive components were then reconceptualized into a proposed construct of pragmatic competence. The challenges of operationalizing pragmatic competence in both instruction and assessment are discussed. The second part of the paper constitutes a domain analysis of pragmatics in the language use domain “workplace.” First, the literature is reviewed for communicative tasks and activities that feature prominently in different workplace settings across various English‐speaking countries. Then, we suggest and exemplify different model task types that can be employed in the context of learning and assessment materials that aim to foster pragmatic‐functional awareness in both English as a foreign language (EFL)/English as a second language (ESL) learners and first language (L1) speakers alike.
One of the most critical steps in the test development process is defining the construct, or the knowledge, skills, or abilities, to be assessed. This foundational step provides the basis for initial assumptions about the meaning of test scores and serves as a reference for subsequent validity research. In this paper, we describe the purpose of the redesigned TOEIC Bridge® 4 skills assessments and elaborate the theoretical basis of its construct definition. We also describe how an evidence‐centered design (ECD) approach was used to develop the redesigned TOEIC Bridge assessments and the first stage of that approach, the domain analysis. The domain analysis begins by elaborating a clearer definition of the context in which language is evaluated by the redesigned TOEIC Bridge assessments, “everyday adult life.” Next, we review research literature and relevant language proficiency standards to highlight the knowledge, skills, and abilities relevant to beginner to low‐intermediate English proficiency for everyday adult life. This information is synthesized in the construct definitions for reading, listening, speaking, and writing ability for beginner to low‐intermediate levels of general English proficiency in the context of everyday adult life.
In this study we examined the extent to which TOEIC® Speaking test scores relate to evaluations by professionals in the international workplace, the target language use domain of TOEIC tests. Linguistic laypersons in 10 countries were invited to participate in an online research survey. The survey incorporated a stratified sample of test-taker ( N = 99) responses to three representative tasks from the TOEIC Speaking test (reading a text aloud, responding to questions, expressing an opinion) that were cast as workplace role-play tasks. After completing each role-play task, participants used brief, descriptive six-point rating scales to rate the communicative effectiveness (comprehensibility, task fulfillment, elaboration, and coherence) of each of several speakers. Communicative effectiveness ratings from linguistic laypersons were strongly correlated with TOEIC Speaking test scaled scores ( r = 0.84). In addition, regression analysis was used to plot the relationship between layperson and test-based evaluations of speaking proficiency. Results suggested that test takers’ performances can be expected to be perceived as effective at score ranges typically associated with important decisions. The results are discussed in terms of their implications for claims about the generalizability of TOEIC Speaking test score interpretations in relation to the evaluations of linguistic laypersons in the international workplace.
The tablet project was undertaken in 2013 to explore the creation of an assessment to reveal English language learners' (ELLs') language capabilities in content area classes such as science. Such assessments could ideally be useful in multiple phases of the instructional process by providing information to learners, teachers, and other stakeholders because the technologies allow for efficiencies of task presentation, response gathering, scoring, storing data, and returning results. With the goal of understanding the development costs required to achieve such benefits, the project used an evidence‐centered design (ECD) framework to create a prototype assessment for ELL middle school students. In addition to describing the ECD process, the report describes how the project mandate helped to focus the complex processes of ECD and how the activities of ECD came into play in sketching plans for the validity argument. The ECD process of domain analysis uncovered perspectives on tablet use in middle schools as well as theoretical perspectives for conceptualizing the tablet users' abilities and learning. Findings included a range of uses of mobile technologies in middle school classrooms in 2013, with little uniformity in practices. Based on analysis of common elements in tablet‐based communication and learning, a framework for characterizing the relevant strategies and abilities was developed. Findings served in the domain modeling process of defining task design patterns, which in turn provided the basis for planning the conceptual assessment framework (CAF). The CAF also drew upon domain analysis findings, which suggested the concepts required for defining a student model (what is to be inferred about the test taker), an evidence model (the basis for making the inference), and the task model (the features of tasks required for eliciting relevant performance). The assessment implementation process produced a working prototype that served in usability testing and gathering feedback from middle school teachers. The report describes how ECD guides the design of tablet‐based formative assessment tasks that take into account the constructs to be measured and the learning that should result from test taking. The evaluation of the project is undertaken in view of the goal of ECD to design test tasks in a manner that serves in the argument for their use. Success is therefore evaluated in part by an analysis of the quality of content it supplied for an interpretation/use argument for the assessment. Overall, the project highlights include (a) the demonstration of how a mandate delimits the ECD processes; (b) the construct framework encompassing both the linguistic and nonlinguistic resources used to create meaning in a digital environment; (c) the approach used for analysis of a domain with emerging, shifting, and variously used technologies; and (d) the use of an interpretation/use argument in the appraisal of an ECD test design.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with 💙 for researchers
Part of the Research Solutions Family.