Design Rationale for a Complex Performance Assessment

Abstract

In computer-based interactive environments meant to support learning, students must bring a wide range of relevant knowledge, skills, and abilities to bear jointly as they solve meaningful problems in a learning domain. To function effectively as an assessment, a computer system must additionally be able to evoke and interpret observable evidence about targeted knowledge in a manner that is principled, defensible, and suited to the purpose at hand (e.g., licensure, achievement testing, coached practice). This article describes the foundations for the design of an interactive computer-based assessment of design, implementation, and troubleshooting in the domain of computer networking. The application is a prototype for assessing these skills as part of an instructional program, as interim practice tests and as chapter or end-of-course assessments. An Evidence Centered Design (ECD)

Interactive computer based assessment tasks: How problem-solving process data can inform instruction

“…This type of evaluation is commonly referred to as Cognitive Task Analysis (CTA) and is typically carried out early in the assessment design process (Mislevy, Steinberg, Breyer, Almond & Johnson, 1999). The function of CTA is to validate the dependency of elicited behaviours upon various structural features of tasks and the knowledge and skills of people undertaking the tasks (Williamson et al, 2004). Importantly in this context, a CTA will reveal important relationships between potentially complex sequences of computercaptured data and corresponding observable variables.…”

“…Recently, computer-based assessments that go beyond being reproductions of existing paper and pencil assessments have emerged in various domains of problem solving (Bennett, Persky, Jenkins & Weiss, 2007;Vendlinski & Stevens, 2002;Williamson et al, 2004, Masters, 2010. These assessments record detailed interactions between the problem solver and the task environment and thereby capture salient solution processes in an unobtrusive way (Bennett et al, 2007;Chung & Baker, 2003).…”

Validating a Mathematics Interim Assessment with Cognitively Diagnostic Error Categories

“…Mislevy's Evidence-Centered Design (ECD) is a detailed, comprehensive assessment design framework involving a series of evidentiary arguments (Behrens, Mislevy, Bauer, Williamson, & Levy, 2004;Gorin, 2007;Mislevy, Almond, & Lukas, 2003;Mislevy & Haertel, 2006;Williamson, Bauer, Steinberg, Mislevy, Behrens, & DeMark, 2004). Mislevy begins by breaking down the test development process into five layers (Mislevy & Haertel, 2006) and four models (Gorin, 2007;Mislevy et al, 2003;Williamson et al, 2004).…”

“…The CAF encompasses the technical specifications, evaluation procedures, and measurement models of the test. The CAF is organized around four models: the student model, evidence model, task model, and the assembly model (Gorin, 2007;Mislevy et al, 2003;Mislevy & Haertel, 2006;Williamson et al, 2004) The four central models of an ECD framework Williamson et al, 2004, p. 306 The student model is represented by a set of claims pertaining to the knowledge, skills, or abilities the test seeks to measure (Gorin, 2007;Mislevy et al, 2003;Mislevy & Haertel, 2006;Williamson et al, 2004). For instance, in the NetPASS computer networking assessment, one of the student model claims was "students can identify the cause of connectivity problems at the physical, data link, and network layers of the OSI mode" (Gorin, 2007, p. 180;Williamson et al, 2004, p. 311) (See Figure 3).…”

“…Evidence could manifest as a log file of IOS commands, configuration files for routers, a worksheet, a network diagram, or an essay (Gorin, 2007, p. 180;Williamson et al, 2004, p. 311). In contrast, the measurement model guides the summary scoring process because it is concerned with the accumulation and synthesis of evidence across tasks (Mislevy et al, 2003;Williamson et al, 2004). Measurement models consist primarily of psychometric models such as classical test theory, item response theory, and cognitive diagnostic models.…”