Improving Construct Validity With Cognitive Psychology Principles

Abstract

Cognitive psychology principles have been heralded as possibly central to construct validity. In this paper, testing practices are examined in three stages: (a) the past, in which the traditional testing research paradigm left little role for cognitive psychology principles, (b) the present, in which testing research is enhanced by cognitive psychology principles, and (c) the future, for which we predict that cognitive psychology's potential will be fully realized through item design. An extended example of item design by cognitive theory is given to illustrate the principles. A spatial ability test that consists of an object assembly task highlights how cognitive design principles can lead to item generation.

Automatic measurement of propositional idea density from part-of-speech tagging

“…In addition, readability-in this very superficial sense-is now commonly computed by word processors and grammar checkers. Automated propositional analyses will open up the possibility of developing style guides for struggling writers as well as applications to critical domains, such as the analysis of text factors affecting health literacy, the improvement of technical documents, and the development and standardization of basal readings and standardized reading assessments (Anderson, 1982;Embretson & Gorin, 2001;Freedle & Kostin, 1991).…”

Cognitive Diagnostic Analysis of Reading Comprehension Items: The Case of English Proficiency Assessment in Saudi Arabia

“…The three most difficult attributes are A3 (ability to describe the writer's purpose), A5 (reasoning), and A6 (evaluation), with some changes in the order of their relative difficulty across the logit scale. For example, the ability to describe the writer's purpose is the most difficult attribute for high ability examinees (about one unit above the average on the logit scale), whereas reasoning is somewhat more difficult for all other examinees [60][61][62][63][64].…”

Using an Instrument Blueprint to Support the Rigorous Development of New Surveys and Assessments in Engineering Education

“…8 In short, score interpretation is dependent upon the validity evidence collected for the instrument itself, making the rigor of the development process for instruments of critical importance. In this section, we present the theoretical framework of the proposed blueprint; we feel the blueprint presented in this work will aid in the development process for instruments by removing the guesswork in the initial phases of instrument creation, as well as bolstering the validity evidence of instruments.…”

“…The next step in item development suggested by the literature 1,6,7,8,9 is the construction of a detailed roadmap of the instrument -i.e., the test specifications. The test specifications lay out a guide to ensure that items are both relevant and representative of the content domain being assessed -i.e., that the items are related to the knowledge, skills, attitudes, and behaviors to be assessed 1 .…”

“… Content Aspect: includes evidence of content relevance, representativeness, and technical quality 1,9  Substantive Aspect: refers to theoretical rationales for the observed consistencies in test responses 1,8  Structural Aspect: appraises the fidelity of the scoring structure to the structure of the construct domain at issue 1,10 Page 26.1657.4…”