Psychometrics in Support of a Valid Assessment of Linguistic Minorities: Implications for the Test and Sampling Designs

Nastal

Slomp

2020

Self Cite

This report discusses frameworks and assessment development approaches to consider fairness, opportunity to learn, and consequences of test use in the design and use of assessments administered to diverse populations. Examples include the integrated design and appraisal framework and the sociocognitively based evidence‐centered design approach. The report also provides an overview of approaches that have been used before to increase the utility, relevance, and authenticity (verisimilitude) of assessments to inform various decisions made from the use of assessments administered to diverse populations. These considerations are important as the test‐taker populations engaging in assessment situations become increasingly more diverse at the national and international levels.

Section: Considerations For Making Assessments More Useful For Diversmentioning

confidence: 99%

Reflections onEquity‐CenteredDesign

Nastal

Slomp

2020

Self Cite

“…The model also aims to help participants identify inconsistencies between national needs and what the ILSA assesses and reports. It is important to identify such inconsistencies to help test users devise plans to supplement ILSA data with other data sources to help mitigate possible unintended consequences of test use, such as unforeseen consequences related to using poorly framed analytical frameworks to analyze ILSA data (Ercikan, ), or unforeseeable consequences related to not acknowledging rapid changes in the makeup of the assessed population (Oliveri & von Davier, ). Identifying the claims that match and those that do not can be a useful practice to help countries better understand, articulate, and prepare for both the intended and unintended consequences of ILSA participation.…”

Section: The Evaluation Modelmentioning

confidence: 99%

Bridging Validity and Evaluation to Match International Large‐Scale Assessment Claims and Country Aims

Rutkowski

2018

Self Cite

Fifty years after the first international large‐scale assessment (ILSA), participation in these studies continues to grow, with more than 50% of the world's countries participating. Concomitant with growth in ILSAs is an expansion in the diversity of participant countries with respect to languages, cultures, and educational perspectives and goals. As educational aims might differ for new participants and goals among historic participants can be expected to shift over time, it is useful to understand the degree to which countries' expectations of ILSAs—as a means for understanding their educational system—align with the explicitly and implicitly stated purposes of these studies. In this presentation, we shift the conversation away from countries reporting ILSA shock and dissatisfaction with participation to fostering a productive conversation about the value and utility of participation. We propose a framework that combines notions from meta‐evaluation to systematically test the evaluation tools—ILSAs and validity theory (in relation to test use and alignment with stakeholder needs) to help countries understand why they participate in ILSAs and the potential value in taking part. We develop this conceptual framework with the aim that countries can (a) systematically consider their educational goals and the degree to which ILSA participation can reasonably help countries monitor progress toward them; (b) use an argument model to analyze claims by ILSA programs against the background of a country's specific context; and (c) more clearly understand intended and unintended consequences of ILSA participation. The framework offers a tool to systematically think through a complex web of implicit and explicit purposes, goals, and actors related to ILSAs and educational systems. To demonstrate our proposed framework, we review national education agendas in several countries with differing educational traditions (e.g., the United States, Mexico, and Norway) against published ILSA frameworks. Using our proposed method would offer a set of general guidelines that national funders can use to chart a path forward in terms of future ILSA participation. It can also equip participating countries with the knowledge to engage in reasoned conversations with testing organizations regarding unmet needs from testing programs.

“…Another approach is to group examinees by language groups. Oliveri and von Davier () exemplified ways to group examinees by language groups, such as Indo‐European versus non‐Indo‐European languages, or combining members of the same language family (e.g., grouping all Romance languages together).…”

Section: Domain Analysis Evaluation and Generalization Componentsmentioning

confidence: 99%

The Validity of Inferences From Locally Developed Assessments Administered Globally

Lawless

2018

Self Cite

In this paper, we first examine the challenges of score comparability associated with the use of assessments that are exported. By exported assessments, we mean assessments that are developed for domestic use and are then administered in other countries in either the same or a different language. Second, we provide suggestions to better support their valid and fair use. We illustrate these issues in the context of higher education assessments that are designed to serve different purposes—inform admissions decisions and assess student learning outcomes within one country (e.g., the United States) that are later used in other countries. In higher education, the use of exported assessments is on the rise due to increases in globalization, student mobility, and cross‐national comparisons of student achievement. An increase in the use of exported assessments leads to more diverse test‐taker populations and requires special attention due to possible sources of construct‐irrelevant variance, which may threaten the score‐based inferences made for various populations. Irrelevant sources of variance may emerge due to differences in opportunity to learn, curricular exposure, and lack of familiarity with the cultural references used in the assessments that are exported.