Threats to validity when using open-ended items in international achievement studies: Coding responses to the PISA 2012 problem-solving test in Finland

Abstract

Open-ended (OE) items are widely used to gather data on student performance in international achievement studies. However, several factors may threaten validity when using such items. This study examined Finnish coders' opinions about threats to validity when coding responses to OE items in the PISA 2012 problem-solving test. Six discussions during six coder practice sessions (on six OE items) and an interview between five coders were audiorecorded and analyzed by means of content analysis. Three main threats to validity were found: (1) unclear and complex questions; (2) arbitrary and illogical coding rubrics; and (3) unclear and ambiguous responses. Suggestions are given as to how to respond to these threats in order to improve the validity of international achievement studies.

Understanding Grader Reliability through the Lens of Cognitive Modeling

“…The meaning and trustworthiness of an assessment score or grade, often referred to as its validity, absolutely requires reliability-that is, the consistency of a score, regardless of when the assessment is conducted, when it is scored, or by whom it is scored [3], [4]. Unfortunately, attaining reliable assessment scores can be challenging in the many engineering courses that utilize open-ended performance tasks to authentically assess the engineering competencies called for by ABET and the Engineer of 2020 [5] - [7]. This challenge increases with larger class sizes, which are often encountered in first-year engineering programs [8].…”