Designing a Prototype Tablet‐Based Learning‐Oriented Assessment for Middle School English Learners: An Evidence‐Centered Design Approach

Abstract

The tablet project was undertaken in 2013 to explore the creation of an assessment to reveal English language learners' (ELLs') language capabilities in content area classes such as science. Such assessments could ideally be useful in multiple phases of the instructional process by providing information to learners, teachers, and other stakeholders because the technologies allow for efficiencies of task presentation, response gathering, scoring, storing data, and returning results. With the goal of understanding the development costs required to achieve such benefits, the project used an evidence‐centered design (ECD) framework to create a prototype assessment for ELL middle school students. In addition to describing the ECD process, the report describes how the project mandate helped to focus the complex processes of ECD and how the activities of ECD came into play in sketching plans for the validity argument. The ECD process of domain analysis uncovered perspectives on tablet use in middle schools as well as theoretical perspectives for conceptualizing the tablet users' abilities and learning. Findings included a range of uses of mobile technologies in middle school classrooms in 2013, with little uniformity in practices. Based on analysis of common elements in tablet‐based communication and learning, a framework for characterizing the relevant strategies and abilities was developed. Findings served in the domain modeling process of defining task design patterns, which in turn provided the basis for planning the conceptual assessment framework (CAF). The CAF also drew upon domain analysis findings, which suggested the concepts required for defining a student model (what is to be inferred about the test taker), an evidence model (the basis for making the inference), and the task model (the features of tasks required for eliciting relevant performance). The assessment implementation process produced a working prototype that served in usability testing and gathering feedback from middle school teachers. The report describes how ECD guides the design of tablet‐based formative assessment tasks that take into account the constructs to be measured and the learning that should result from test taking. The evaluation of the project is undertaken in view of the goal of ECD to design test tasks in a manner that serves in the argument for their use. Success is therefore evaluated in part by an analysis of the quality of content it supplied for an interpretation/use argument for the assessment. Overall, the project highlights include (a) the demonstration of how a mandate delimits the ECD processes; (b) the construct framework encompassing both the linguistic and nonlinguistic resources used to create meaning in a digital environment; (c) the approach used for analysis of a domain with emerging, shifting, and variously used technologies; and (d) the use of an interpretation/use argument in the appraisal of an ECD test design.

Designing proficiency-oriented performance tasks for the 21st-century workplace written communication: An evidence-centered design approach

“…ECD does not provide a prescriptive methodology or a list of components that should be analyzed. What needs to be investigated and how it should be analyzed depends on the particular test development project (Chapelle et al, 2018). That is, each project needs to define its own scope and breadth (i.e.…”

“…which skills were in more or less demand and how those could be represented in assessment tasks). More importantly, ECD can help create a solid argument for validation of test score interpretation and use by establishing explicit links between design decisions and scores obtained from the test (Chapelle et al, 2018). For CEPA, the ECD process was used to define and operationalize the construct of writing in the context of workplace communication leading to the design and development of the CEPA Written Communication Assessment, the outline for which is further exhibited in Figure 1.…”

“…As previously mentioned, the ECD framework, as the first phase, requires a detailed analysis of the target language use domain for which the test is developed, and the scores used. The identification and analysis of the various components in a given domain depends on the particular test development project (Chapelle et al, 2018). With this understanding, the components (i.e.…”

“…Schematic diagram of inputs, activities, and results for domain analysis. Portions of this figure were adapted fromChapelle et, al. (2018)…”

Utilizing Evidence-Centered Design to Develop Assessments: A High School Introductory Computer Science Course

“…In an attempt to optimize this key piece of CS assessment, we adopted the Evidence-Centered Design (ECD) framework to guide our assessment development efforts. ECD has been used effectively for assessment development in various fields (Grover and Basu, 2017;Hu et al, 2017;Chapelle et al, 2018;Bechard et al, 2019;Oliveri et al, 2019;Snow et al, 2019); it is especially useful for starting assessment development off on the right foot by virtue of its rigorous and scaffolded methodology for clearly defining the content to be measured and what it means for students to engage with this content (Mislevy and Riconscente, 2006). The ECD framework at its core helps assessment developers articulate exactly what is to be measured and what evidence is needed to measure it and provides support to build the argument that the assessment meets its specified purpose (Mislevy, 2007).…”