The promise of open survey questions—The validation of text-based job satisfaction measures

Abstract

Recent advances in computer-aided text analysis (CATA) have allowed organizational scientists to construct reliable and convenient measures from open texts. As yet, there is a lack of research into using CATA to analyze responses to open survey questions and constructing text-based measures of psychological constructs. In our study, we demonstrated the potential of CATA methods for the construction of text-based job satisfaction measures based on responses to a completely open and semi-open question. To do this, we employed three sentiment analysis techniques: Linguistic Inquiry and Word Count 2015, SentimentR and SentiStrength, and quantified the forms of measurement error they introduced: specific factor error, algorithm error and transient error. We conducted an initial test of the text-based measures’ validity, assessing their convergence with closed-question job satisfaction measures. We adopted a time-lagged survey design (Nwave 1 = 996; Nwave 2 = 116) to test our hypotheses. In line with our hypotheses, we found that specific factor error is higher in the open question text-based measure than in the semi-open question text-based measure. As expected, algorithm error was substantial for both the open and semi-open question text-based measures. Transient error in the text-based measures was higher than expected, as it generally exceeded the transient error in the human-coded and the closed job satisfaction question measures. Our initial test of convergent and discriminant validity indicated that the semi-open question text-based measure is especially suitable for measuring job satisfaction. Our article ends with a discussion of limitations and an agenda for future research.

Unpacking the Quantifying and Qualifying Potential of Semi-Open Job Satisfaction Questions through Computer-Aided Sentiment Analysis

“…Much of the research on constructing text measures from responses to semi-open or open job satisfaction questions have made use of computer-aided sentiment analysis techniques. The techniques' popularity is not surprising, as it is much faster than manual sentiment annotation (Wijngaards et al 2019) and an individual's choice of words is a plausible manifestation of thoughts and opinions (Pennebaker et al 2003;Short et al 2010). As mentioned earlier, computer-aided sentiment analysis is particularly suitable for constructing a job satisfaction measure from text, as job satisfaction classifies as a job attitude and comprises cognitive appraisals, emotions, and beliefs (Weiss 2002).…”

“…To illustrate this two-stage procedure, we use a response from a semi-open question and a response to an open job satisfaction question, both obtained from the open-access data from Wijngaards et al (2019). The sample answer to the semi-open question was a list of three adjectives: "Interesting", "Stressful" and "Helpful".…”

“…In this article, we aim to unpack the quantifying and qualifying potential of a semi-open job satisfaction question. We focus on a semi-open rather than a completely open job satisfaction question, because semi-open questions impose answering constraints on responses and therefore produce more structured, e.g., fewer meaningless words and less semantic nuance, than completely open questions (Glerum et al 2014;Wijngaards et al 2019). This makes text measures based semi-open questions produced by computer-aided sentiment analysis methods probably more suitable for quantifying the level of job satisfaction than text measures based on completely open questions produced by computer-aided sentiment analysis methods (Wijngaards et al 2019).…”

“…We focus on a semi-open rather than a completely open job satisfaction question, because semi-open questions impose answering constraints on responses and therefore produce more structured, e.g., fewer meaningless words and less semantic nuance, than completely open questions (Glerum et al 2014;Wijngaards et al 2019). This makes text measures based semi-open questions produced by computer-aided sentiment analysis methods probably more suitable for quantifying the level of job satisfaction than text measures based on completely open questions produced by computer-aided sentiment analysis methods (Wijngaards et al 2019). We investigate the semi-open questions' quantifying potential by creating text measures using computer-aided sentiment analysis, the practice of automatically detecting opinions, sentiments, attitudes and emotions about certain objects in humangenerated texts (Feldman 2013;Liu 2015), and validating this measure using well-established survey scales.…”

“…This study contributes to the literature by building on and extending existing methodological work on validation of textual job satisfaction measures. Previous studies on textual job satisfaction measures lack a systematic validation approach, neglecting content validity and using ad-hoc, single-item job satisfaction measures to test convergent validity (Borg and Zuell 2012;Poncheri et al 2008;Wijngaards et al 2019), as well as discriminant validity (Borg and Zuell 2012;Gilles et al 2017;Poncheri et al 2008). It is essential to systematically examine the validity of a text measure, as researchers with a preference for traditional survey measures are unlikely to accept text measures as fruitful complement to a closed question if there is no convincing evidence for their validity available.…”