8. A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design

Abstract

z Two distinctly different quantitative approaches are used to evaluate measurement instruments: the split-ballot experiment and the multitrait-multimethod (MTMM) approach. The first approach is typically used to indicate whether variation in the method causes differences in the response distribution; the second approach evaluates the reliability and validity of different methods. The new approach, suggested in this paper, combines the more attractive features of both methods. The strength of the split-ballot experiment is its use of independent random samples from the same population to provide information about differences in response distributions. This is also possible with the new approach, but this approach provides more detailed information about the reasons We thank the anonymous reviewers for their comments and the support of the Spanish Ministry of Science and Technology through the grant SEC2003-04476.

Estrategias para aumentar la tasa de respuesta y los resultados de la Encuesta Social Europea en España

“…Como es bien sabido, la manera como están formuladas las preguntas, las opciones de respuesta disponibles y su estructura influyen también en la calidad de la encuesta. Existe abundante literatura al respecto y se han realizado múltiples experimentos que muestran cómo el resultado a la pregunta sobre un determinado tema varía en función del instrumento de medida utilizado, es decir del ítem o pregunta de la encuesta formulada (véase Saris y Munich 1995; Saris et al 2004;Saris y Gallopher 2007). También se encuentra abundante literatura sobre los sesgos que puede introducir el entrevistador en las respuestas obtenidas en una entrevista cara a cara (Briemer y Lyberg 2003).…”

Measuring Left-Right Political Orientation: The Choice of Response Format

“…For the estimation of measurement error, Saris et al (2004) suggest a design that combines two of the classic approaches: an experimental design and statistical modeling. The multi-trait multi-method (MTMM) approach was suggested first by Campbell and Fiske in 1959 and has since then attracted much attention in survey research (for an overview, see Wothke 1996).…”

“…7 Even though not all combinations of traits and methods are observed for all respondents, validity and reliability parameters can nonetheless be identified by normal theory maximum likelihood in multiple groups, assuming a common model, i.e. with equality constraints of all parameters across random groups (for a discussion of the estimation technique see Saris et al 2004). …”

Design And Analysis Of Incomplete Multitrait-Multimethod Studies From A Multiplicative Perspective

“…The complete MTMM design requires mt measurements from each subject. This requirement may be too restrictive and is unnecessary (Marsh, 1990;Saris, Satorra, & Coenders, 2004). An MTMM design is called an incomplete MTMM design if some of its 362 PSYCHOMETRIKA trait-method combinations (TMCs) are missing.…”