Experimental Test Validation

Abstract

Abstract. Although the vast majority of validation studies rely on correlational validity evidence, there is an increased recognition that validity should also focus on whether variations in the focal psychological attribute lead to variations in the measurement outcomes. Therefore, calls have been made that validity evidence should also be gathered through experiments. Existing experimental validation strategies focus on manipulating psychological attributes and their effects on measurement outcomes. In the current manuscript, we present an additional and complementary approach that focuses on manipulating test elements (instead of psychological attributes) that are considered indispensable for test functioning. Examples from personality, situational judgment, emotional intelligence, and reading comprehension domains are presented to illustrate our approach. The presented approach is integrated into existing validation strategies.

Unveiling an exclusive link: Predicting behavior with personality, situation perception, and affect in a preregistered experience sampling study.

“…This may, for example, be possible in some professional contexts (e.g., counseling, where the consultant perceives that negative feelings are present, while at the same time experiences no personal negative affect). The successful discriminant experimental manipulation of a scale score can therefore be seen as further evidence for its (discriminant) validity (Borsboom, Mellenbergh, & van Heerden, 2004; Krumm, Hüffmeier, & Lievens, 2019). Thus, the current findings should be regarded as encouraging evidence supporting the notion of specific relations between the constructs assessed.…”

Towards a Process Model of Sustained Attention Tests

“…Tests of sustained attention (sometimes also referred to as concentration) are among the most popular psychological tests in the domain of cognitive abilities and their validity has been demonstrated in numerous studies [1,2,3,4,5]. Yet, as the field of psychological assessment progresses, calls have been made to take a closer look into the processes that underly test scores and gain a deeper understanding of how these tests scores emerge [6,7,8,9]. For example, the new Standards for Educational and Psychological Testing request—in their first chapter—that validity evidence should not only be provided in terms of the test’s internal structure or its relation to other measures, but also in terms of the involved response processes [10].…”

Arousing Motives or Eliciting Stories? On the Role of Pictures in a Picture–Story Exercise

“…To examine the role of pictures in PSEs, the present study adopted an experimental test validation approach (Bornstein, 2011; Borsboom et al, 2004; Erdfelder & Musch, 2006; Krumm et al, 2017). That is, we manipulated a crucial element of a test and examined whether it made a difference for test results.…”