Integrative Complexity Coding Raises Integratively Complex Issues

Abstract

and Houck (2014) report a major effort to automate integrative complexity coding. Judging this effort requires researchers to be more explicit in articulating key methodological assumptions about the coding process and theoretical assumptions about the construct. Unresolved issues include: (1) when, and on what basis, we should attribute divergences between human coders and algorithms to overestimations or underestimations by one or the other approach; and (2) to what extent second-generation algorithms can yield Pareto improvements that reduce errors of both underestimation and overestimation. Further progress in developing natural language processing measures of this cognitive style will require sharper definitions of target constructs: in particular, different types of differentiation (dialectical and elaborative) and integration (hierarchical and flexible) and clearer guidelines for factoring context into assessments.

Integrative Complexity at Forty: Steps Toward Resolving the Scoring Dilemma

“…These include the comparative levels of disagreement among human scorers and between human and computer scoring: passages on which human scorers disagree with each other more should also show more disagreement between human and computer scoring. For reasons elaborated in the article by Tetlock, Metz, Scott, and Suedfeld (), human scorers usually show high agreement in the lower ranges of IC but diverge when they come to passages in the 5–7 range. Similarly, human‐computer correlations at the higher levels tend to nose–dive, sometimes close to zero (cf.…”

“…Tetlock et al. () suggest three possible “ultimate arbiters,” but only the coder as arbiter seems convincing; and, as the article indicates, there is considerable scope for both over‐ and underestimation of IC by both humans and computer algorithms. Replicating the nomological net, also advocated in the article, can test construct or concurrent validity.…”

Increased Complexity Has Its Benefits

“…Many of the obstacles to the development of the more robust “second generation” automated IC coding schemes called for by Tetlock, Metz, Scott, and Suedfeld () can be completely or partially overcome by adopting a multiple‐pass text‐analysis system. A number of these systems, such as GATE, VisualText, and Profiler Plus, are available at low or no cost (see Table ).…”

“…As Tetlock et al. () suggest, a straightforward way to approach such a concern is with sentiment analysis, an idea that can be implemented simply with lists of “good” and “bad” valued words. “Best,” “wisdom,” and “funny” are all “good” valued words while “worst,” “foolishness,” and “bitter” are “bad.” In a first pass through the text, words could be coded as “good” or “bad” and then, in a second pass, clauses or sentences that contained both “good” and “bad” would be marked as indicating complexity, or in the case of IC, as indicating differentiation.…”

Automated Integrative Complexity: Current Challenges and Future Directions

“…Automating integrative complexity is fraught with many challenges. We would first like to express our sincere gratitude to Suedfeld and Tetlock (), Tetlock, Metz, Scott, and Suedfeld (), and Young and Hermann () for tackling some of these challenges. Their contributions in this symposium spur an important dialogue about the current state of automating complexity and its future possibilities.…”