The Validity and Reliability of Survey Questions

Abstract

Inspired by the research of Frank Andrews on the reliability and validity of survey questions, a large-scale research project was conducted in the Netherlands. The project was comprised of two different stages. For this project, more than 600 survey questions were included in different surveys according to a multitrait-multimethod design. The resulting data were analyzed in two steps. In the first step, estimates of validity and reliability were obtained for each question. The second step was a meta-analysis of the variation in data quality found in the first step. This variation was related to question-specific characteristics, response scale characteristics, context characteristics, and design characteristics. The article describes how the results of this study can be of practical use. In addition, the authors compare them to results of similar studies in the United States, Austria, and other Western, Central, and Eastern European countries.

8. A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design

“…If enough MTMM experiments are carried out and a meta-analysis to determine the effects of choices on quality criteria is undertaken, then no extra questions are needed in the substantive surveys. This approach is indeed what has been suggested by Andrews (1984) and is also applied in several other studies (Ko¨ltringer 1995;Scherpenzeel and Saris 1997;Saris and Gallhofer, forthcoming).…”

“…However, an alternative to such studies is the use of meta analysis of many separate MTMM experiments under different conditions. This approach was suggested by Andrews (1984), was further explored by Saris and Mu¨nnich (1995), and was applied by Rodgers, Andrews, and Herzog (1992), Ko¨ltringer (1995), Scherpenzeel (1995), and Scherpenzeel and Saris (1997).…”

Measuring Left-Right Political Orientation: The Choice of Response Format

“…This does of course not imply that the 11-point scale is the optimal response format for measuring attitudes under all circumstances. Rather, the appropriateness of a survey measure is a function of characteristics of the concept under study, such as domain and complexity, characteristics of respondents, such as their familiarity with the concept, and characteristics of the interview setting, such as mode of data collection (e.g., Andrews 1984;Alwin and Krosnick 1991;Scherpenzeel and Saris 1997).…”

Issues of Adaptive Automated Surveys in a Computer Network Environment

“…A number of studies (e.g., Koltringer, 1995;Rodgers, Andrews, & Herzog, 1992;Scherpenzeel & Saris, 1993;1997) have examined a multitude of different cognitive psychological issues across the various survey administration types in terms of issues of reliability and validity. Many of the findings from these studies should generalize to computer network surveys.…”