Semantic algorithms can detect how media language shapes survey responses in organizational behaviour

Abstract

Research on sensemaking in organisations and on linguistic relativity suggests that speakers of the same language may use this language in different ways to construct social realities at work. We apply a semantic theory of survey response (STSR) to explore such differences in quantitative survey research. Using text analysis algorithms, we have studied how language from three media domains–the business press, PR Newswire and general newspapers–has differential explanatory value for analysing survey responses in leadership research. We projected well-known surveys measuring leadership, motivation and outcomes into large text samples from these three media domains significantly different impacts on survey responses. Business press language was best in explaining leadership-related items, PR language best at explaining organizational results and “ordinary” newspaper language seemed to explain the relationship among motivation items. These findings shed light on how different public arenas construct organizational realities in different ways, and how these differences have consequences on methodology in research on leadership.

The Priest, the Sex Worker, and the CEO: Measuring Motivation by Job Type

“…A third possibility would be systematic differences in the way items are processed (cf. Arnulf et al, 2018c), which is what we are really looking for here. Our data show signs of all three explanations.…”

Culture Blind Leadership Research: How Semantically Determined Survey Data May Fail to Detect Cultural Differences

“…The first way to test this is to see how well the semantically predicted correlations actually match the real survey correlations. Central to leadership research is an interest in the mutual impact of leadership behaviors on purported outcomes (March and Sutton, 1997;Dumdum et al, 2002;Hansen et al, 2013;Arnulf et al, 2018d). Since the MLQ contains a separate scale for outcomes, we can average the correlations between each leadership behavior and the outcome measures and compare these to the values predicted in the respective regression models.…”

“…So far, we know that survey structures vary between almost complete semantic predictability to almost nothing at all (as in the case of the NEO personality inventory) (Arnulf et al, 2014a). It is likely that the phenomenon is more prevalent where the measures are reflective and the latent variables are social constructions (Arnulf et al, 2018d) than if the measures are formative (Arnulf, 2020). Several studies are going on to determine the variance components most influential in shaping semantic patterns, among others by applying multitrait-multi-method (MTMM) approaches (Martinsen et al, 2017) but the picture is not yet conclusive.…”

“…The word "leadership" does not exist in, e.g., French, Italian or Japanese, but are usually substituted with the English word. The German counterpart for leadership ("Führung") was politically contaminated and has largely been replaced with the English word "Management" (Arnulf et al, 2018d), but with slightly different meanings -what the linguists call "false friends" (Enfield, 2007).…”

“…Many Norwegians will actually know of the English idiom, but a search for the phrase at Google.no will yield articles literally about whether consumers buy books based on the attractiveness of the cover. The requirement of idiomatic equivalence is common knowledge to most translators but it bears special relevance to the problem of semantic determination of survey response statistics (Arnulf et al, 2018d). If the translation departs from the idiographic essence, it can be inaccurate even when the superficial words look similar.…”