Short, Simple, and Specific: The Influence of Item Design Characteristics in Multi‐Source Assessment Contexts

Abstract

The present study investigates the impact of item characteristics on multi-source performance assessment. Three item characteristics (syntax, double-barreledness, & behavioral specificity) were linked to the psychometric properties of items used by self, subordinates, peers, and supervisors as operationalized by the relationship between the item and the performance dimension it is intended to measure. Results show that syntax, a linguistic index that pertains to the length of items, is related to the psychometric properties of all rating sources except subordinates. The implications of this effect for the design of multi-source assessment instrument are discussed.

Do Raters Agree More on Observable Items?

“…The average r wg for the expert raters across all 86 item was 0.66 (SD = 0.27), close to the .70 level and is very similar to the agreement levels found by both Brutus and Facteau (2003) and Kaiser and Craig (2005), r wg = .64 and r wg = .68, respectively, for their expert raters. Furthermore, in the current study the experts had an r wg of greater than .80 on 42 of the 86 items.…”

“…Kaiser and Craig (2005) adopted Brutus and Facteau's (2003) definition of specificity and defined it in terms of several dimensions, an observable and narrow definition of the behavior and the contextual frame in which it occurs (i.e., not specific at all to extremely specific). In contrast to these previous studies, we focused only on one dimension-amount of judgment needed to determine whether the behavior occurred.…”

“…We wished to directly investigate this hypothesis, but on the item level instead of the dimension level. It should be noted that this definition of behavioral observability differs from the definition of behavioral specificity used by Brutus and Facteau (2003) and Kaiser and Craig (2005), who defined specificity in terms of both a narrow definition BEHAVIORAL OBSERVABILITY Note. Observability ratings are reverse scored; higher numbers denote more observable items.…”

“…Recently, researchers have begun to empirically investigate how item characteristics influence ratings (Brutus & Facteau, 2003;Kaiser & Craig, 2005). Even though Brutus and Facteau did not investigate rater agreement, reliability, or to what extent the items are observable, they did investigate behavioral specificity, which is closely related to behavioral observability.…”

Improving Feedback Reports: The Role of Procedural Information and Information Specificity

“…In response to this call, researchers have recently started to investigate the effects of type, amount and specificity of the information provided in feedback messages on performance and other feedback-related outcomes. For instance, the use of numeric, normative, or text feedback (Atwater & Brett, 2006), the specificity of the feedback presented (Goodman & Wood, 2004b), the amount of comments and whether they contain behavior-or task-focused information (Smither & Walker, 2004) and whether feedback is precise (Brutus, 2009;Brutus & Facteau, 2003) are all characteristics that have been found to be important in determining outcomes of feedback and performance improvement. In a recent review, Brutus (2009) (Brutus, 2009: 11).…”