Validating automated integrative complexity: Natural language processing and the Donald Trump Test

Abstract

Computer algorithms that analyze language (natural language processing systems) have seen a great increase in usage recently. While use of these systems to score key constructs in social and political psychology has many advantages, it is also dangerous if we do not fully evaluate the validity of these systems. In the present article, we evaluate a natural language processing system for one particular construct that has implications for solving key societal issues: Integrative complexity. We first review the growing body of evidence for the validity of the Automated Integrative Complexity (AutoIC) method for computer-scoring integrative complexity. We then provide five new validity tests: AutoIC successfully distinguished fourteen classic philosophic works from a large sample of both lay populations and political leaders (Test 1) and further distinguished classic philosophic works from the rhetoric of Donald Trump at higher rates than an alternative system (Test 2). Additionally, AutoIC successfully replicated key findings from the hand-scored IC literature on smoking cessation (Test 3), U.S. Presidents’ State of the Union Speeches (Test 4), and the ideology-complexity relationship (Test 5). Taken in total, this large body of evidence not only suggests that AutoIC is a valid system for scoring integrative complexity, but it also reveals important theory-building insights into key issues at the intersection of social and political psychology (health, leadership, and ideology). We close by discussing the broader contributions of the present validity tests to our understanding of issues vital to natural language processing.

Natural Language Analysis and the Psychology of Verbal Behavior: The Past, Present, and Future States of the Field

“… 3 We note that while the “words as attention” interpretation of verbal behavior is implicit to most mainstream language analysis research today, it is not the only interpretation (and not even necessarily the only correct interpretation). Other conceptualizations of language approach verbal behavior as tapping into the degree or orientation of dimensional psychological processes, typologies that reflect psychodynamic “modes” of thinking, or even stages of psychosocial development (see, e.g., Bucci & Maskit, 2006; Conway et al, 2020; Lanning et al, 2018). Nevertheless, the majority of mainstream language analysis frameworks can be seen to rely on the implicit assumption of “words as attention” if we dig deep enough. …”

In Search of the Cognitively Complex Person: Is There a Meaningful Trait Component of Cognitive Complexity?

“…This person effect represents the stability of individual levels of integrative complexity over time (see Thoemmes & Conway, 2007; see also Wasike, 2017). This person effect was recently replicated using AutoIC on U.S. presidents’ State of the Union speeches (Conway et al, 2020), showing a similarly small-but-significant effect, estimated r (18, 848) = .02, p < .001. 8…”

“…By far the single best set of individual difference predictors for 40 U.S. presidents’ integrative complexity (scored using updated guidelines by Baker-Brown et al, 1992) were those related to social variables: affiliation motive, r (39) = .40, p < .05, extraversion, r (39) = .36, p < .05, friendliness, r (39) = .32, p < .06, and wittiness, r (39) = .42, p < .05 (Thoemmes & Conway, 2007). These effects were recently partially replicated using a larger data set scored for AutoIC for State of the Union speeches (Conway et al, 2020): Affiliation motive showed a significant relationship to integrative complexity, r (39) = .35, p < .05, although relationships for friendliness, r (39) = .12, and wittiness, r (39) = .24, were weaker and nonsignificant, and extraversion was actually mildly negatively (though nonsignificantly) related, r (39) = −.08. Wasike (2017) additionally found that U.S. presidents’ integrative complexity (scored using updated guidelines by Baker-Brown et al, 1992) was associated with the social trait charisma (see Table 2 for specifics on this association).…”

“…These completed statements were scored for integrative complexity by the Automated Integrative Complexity scoring system 5 (AutoIC; Conway et al, 2014; C. C. Houck et al, 2014 for updated validity evidence for AutoIC, see Conway et al, 2020). Individuals’ integrative complexity scores on the two different topics were correlated at r (200) = .41, p < .001.…”

“…Unfortunately, SEM on trait, state, and error variance has yet to be applied to cognitive complexity data, and indeed, generally speaking, the available data would not meet the requirements for running such analyses (as outlined by Kenny & Zautra, 1995). This is undoubtedly in part because measuring cognitive complexity output has historically been a very time-consuming endeavor that requires much more investment per participant than traditional self-report or observational scales (see Conway et al, 2014, 2020). The evidence currently available for the across-time stability of cognitive complexity utilizes the first method described above: correlating individuals’ state cognitive complexity across differing time points or contexts using either raw scores of cognitive complexity or rank-ordered scores of participants’ cognitive complexity.…”