Information Entropy and Scale Development

Abstract

A wide variety of techniques are used to assess the development of survey-based scales. The majority of these techniques focus on the quality of information characterized by the scale. Aside from very rudimentary measures such as response rates and sample sizes, very few empirical techniques are available to measure the quantity of information contained in a scale. This article conducts an exploratory empirical analysis to assess whether information entropy can be useful for measuring the quantity of information in a scale’s development. If the quantity of information in the scale significantly increases (decreases) with the addition of the survey item, researchers may consider retaining (discarding) that item in the scale. The study was conducted within the context of a natural experiment that occurred at a major amateur sporting event in 2018. Customer satisfaction was assessed using a survey whose core questions have been assessed repeatedly over time. The most recent survey contained a previously validated empathy scale, with two items contained in the base measure. Six additional items were added to this base empathy measure. The quantity of information provided (as measured by information entropy) is calculated for each set of scale items. Statistical analysis indicates that, when adding the behavioral, cognitive, and affective scales to the two-item base scale, the quantity of information available increased. However, most of the increase in information quantity was attributable to three survey items, one item from each of the behavioral, cognitive, and affective domains. These findings suggest that information entropy may indeed be a useful quality control tool for survey scale development.

Information Entropy as a Quality Control Tool in Survey Research

“…In other words, the distribution of survey responses is expected to be uniform. A uniformly distributed response minimizes the likelihood of leniency, common method variance and/or framing biases (among other design issues) which reduce the sensitivity of the survey item(s) or scale(s) being analyzed (Friesner et al, 2016;Smith & Albaum, 2005). Movement away from a uniform distribution also implies that respondents are providing a unique quantity of information in their responses that cannot be obtained through "statistical chance," or more specifically the assignment of responses based on a uniform distribution.…”

“…Extensions of the basic information entropy measure described by (1) have been proposed in literature (Dahl & Osteras, 2010;Friesner et al, 2013;Friesner et al, 2016;Friesner et al, 2021;Friesner et al, 2022) to address practical considerations. For instance, a typical survey might ask an individual to respond to several survey items.…”

“…Comparing entropy values across survey items allowed for a relative assessment of which survey items contain a greater quantity of information than others, and by extension provided inferences about which student ratings of instruction questions are more informative than others. Friesner et al (2022) apply information entropy to determine whether the same survey items, when administered repeatedly over time to similar populations, provide varying quantities of information. They found that the quantity of information increased slightly over repeated survey administrations.…”

“…This indicates that valid and reliable surveys, when administered repeatedly, should exhibit "dynamic stability." Friesner et al (2021) develop a methodology to evaluate whether the inclusion or exclusion of a survey item in a construct changes the quantity of information provided by respondents. This gives researchers an additional tool to help ensure that any set of survey items comprising a construct are designed efficiently (i.e., a smaller number of survey items can be used to accurately and precisely characterize a construct).…”

“…An alternative approach was posited by Friesner et al (2016), who (within the context of a single survey administration) examined the distribution of responses to a series of survey questions (which presumably measure the same underlying construct) provided by a single respondent. This facilitates relative assessments about which groups of respondents provide relatively more or less information in their responses for that construct.…”