Inter-rater reliability for movement pattern analysis (MPA): measuring patterning of behaviors versus discrete behavior counts as indicators of decision-making style

Abstract

The unique yield of collecting observational data on human movement has received increasing attention in a number of domains, including the study of decision-making style. As such, interest has grown in the nuances of core methodological issues, including the best ways of assessing inter-rater reliability. In this paper we focus on one key topic – the distinction between establishing reliability for the patterning of behaviors as opposed to the computation of raw counts – and suggest that reliability for each be compared empirically rather than determined a priori. We illustrate by assessing inter-rater reliability for key outcome measures derived from movement pattern analysis (MPA), an observational methodology that records body movements as indicators of decision-making style with demonstrated predictive validity. While reliability ranged from moderate to good for raw counts of behaviors reflecting each of two Overall Factors generated within MPA (Assertion and Perspective), inter-rater reliability for patterning (proportional indicators of each factor) was significantly higher and excellent (ICC = 0.89). Furthermore, patterning, as compared to raw counts, provided better prediction of observable decision-making process assessed in the laboratory. These analyses support the utility of using an empirical approach to inform the consideration of measuring patterning versus discrete behavioral counts of behaviors when determining inter-rater reliability of observable behavior. They also speak to the substantial reliability that may be achieved via application of theoretically grounded observational systems such as MPA that reveal thinking and action motivations via visible movement patterns.

Decision-Making Style in Leaders: Uncovering Cognitive Motivation Using Signature Movement Patterns

“…Those who relied more upon Perspective as opposed to Assertion reached for more information before arriving at a decision, and had overall longer reaction times. Further work revealed that the type of patterning that is coded in MPA-the individually-referenced balance between Assertion and Perspective-is a superior metric for predicting decision-making process, as compared to raw counts of signature movements representing Assertion and Perspective (Connors et al, 2014). The current work linking MPA to observable decision-making process converges with recent studies, reviewed above, on individual differences in decision making, in two regards.…”

“…Furthermore, recent experimental work has demonstrated that MPA is predictive of decision-making process as elicited in the laboratory (Connors et al, 2013;Connors et al, 2014). A sample of military leaders were shown to vary substantially in the way in which they balanced the need for Assertion and Perspective, based upon their MPA profile.…”

“…Particular attention has been directed toward the notable differences in decision-making style that can be observed when studying a group of leaders (see Connors, Rende, & Colton, 2013;Connors, Rende, & Colton, 2014). For example, prior studies have shown notable variation in decision-making style in experienced leaders in general (Carnevale, Inbar, & Lerner, 2011), leaders across different types of professional background (Laureiro-Martinez et al, 2014), and military officers (Thunholm, 2004).…”

“…The core idea is that individuals need to balance multiple actions/motivations throughout stages of decision making (see Connors et al, 2013;Connors et al, 2014). Assertion reflects the exertion of tangible energy (physical and mental) in the environment in order to get results and make things happen.…”

“…In this paper, we focus on Movement Pattern Analysis (MPA), a theory-driven observational methodology which has been used for decades in business to guide executive recruitment, position selection, and the building of management teams, and has been applied in physical and psychological therapies (see Lamb, 2012;Moore, 2005). Briefly, MPA centers on the recording and coding of a number of "posture-gesture mergers" (PGMs), which are observable behaviors that are considered to correspond to multiple stages of the decision-making process (see Connors et al, 2013;Connors et al, 2014, for more detailed description).…”