Split Questionnaire Design for Massive Surveys

Abstract

Instead of the heuristic randomization methods to design split questionnaires that are currently used in applied and academic research, we develop a methodology to design the split questionnaire to minimize information loss using estimates from a first wave or pilot study. Because the number of possible questionnaire designs is exponential in the number of questions, we apply the Modified Federov algorithm, using Kullback Leibler Distance as a design criterion, to find the optimal splits. We use Markov chain Monte Carlo procedures to impute missing values that result from the design. First of all, we illustrate the efficiency of the Modified Federov Algorithm on a small synthetic questionnaire, which enables the enumeration of all possible designs for comparison. Second, we compare the efficiency of split questionnaires generated with the proposed method to multiple matrix sampling, incomplete block designs, and a heuristic procedure based on principal components analysis, using synthetic and empirical Web survey data. We generate split questionnaire designs selecting either entire blocks of questions (between-block design) or sets of questions in each block (within-block design). Finally, we illustrate that due to reduced respondent burden the quality of data using split designs increases, compared to a full questionnaire in a field study.

An Efficient Method for Estimating Population Parameters Using Split Questionnaire Design

“…However, this is out of our scope due to the logistic issue. This subject has been adressed by Adigüzel and Wedel (2008) who construct a field study to …”

“…In addition, a fixed set of items from the original questionnaire is considered to be asked from all sample units. Later, Adigüzel and Wedel (2008) proposed a strategy to design optimal split questionnaire surveys by applying the Kullback-Leibler distance (Kullback and Leibler, 1951). In other words, their proposed criterion to find the optimal design is to minimize the information loss compared to the original questionnaire.…”

Real-time customer probing and decision support in a call center

“…In the marketing science community, there is a body of related research dealing with split survey design [15], which is concerned with scientifically breaking up a survey into smaller pieces and administering each part to a randomly selected group of respondents. In a split-survey, the survey is designed before-hand, and there is no prior data assumed to be available about the respondent.…”

Factor Structure and Measurement Invariance of the Self-discipline Model Using the Different-length Questionnaires: Application of Multiple Matrix Sampling

“…[3]; 4) the return of short-length questionnaire with the probability is greater than the return of the long-length questionnaire ( [17]; [33]); 5) the length of the questionnaire correlates with the intention to read and respond to the questionnaire completely; 6) the respondents are more likely to misrepresent the factual information and have a large amount of missing data when using long-length questionnaire. There are also opportunities for rejection in the entire questionnaire ( [28]; [10]; [2]). Therefore, the longer the questionnaire is, more possible for missing of fact and more difficult to solve problems with statistical procedures (such as missing value replacement, converting data to quality based on statistical usage conditions).…”

“…Consequently, it is necessary to use the questions to be as representatives measuring the sufficiently covered content, such as TIMSS, PISA, PIRLS, NAEP ( [10]; [13]). Even the MMS technique has the advantage of collecting social data in the past, the application is still in the scope of massive survey [2] and large scale measurements of what the researchers are interested in. ( [13]; [43]; [10]; [20]).…”