The How of Survey Self-report: VAS-Likert-Slide-Swipe... Same difference?

Abstract

Self-report is a fundamental research tool for the social sciences. Despite quantitative surveys being the workhorses of the self-report stable, few researchers question their format-often blindly using some form of Labelled Categorical Scale (Likert-type). This study presents a brief review of the current literature examining the efficacy of survey formats, addressing longstanding paper-based concerns and more recent issues raised by computer-and mobile-based surveys. An experiment comparing four survey formats on touch-based devices was conducted. Differences in means, predictive validity, time to complete and centrality were compared. A range of preliminary findings emphasise the similarities and striking differences between these self-report formats. Key conclusions include: A) that the two continuous interfaces (Slide & Swipe) yielded the most robust data for predictive modelling; B) that future research with touch self-report interfaces can set aside the VAS format; C) that researchers seeking to improve on Likert-type formats need to focus on user interfaces that are quick/simple to use. Implications and future directions for research in this area are discussed.

A Proposed Remedy for Grievances about Self-Report Methodologies

“…Durik and Jenkins (2020) administered survey items calling for respondents to the degree to which they agreed with statements describing interest in several areas of study: astronomy, biology, math and psychology; and a single item inviting respondents to classify how certain they were about the set of ratings of interest. Fryer and Nakao (2020) investigated several judgments students made about a course. Topics ranged over interest, meaningfulness and other dimensions.…”

“…Other assumptions, usually unstated and commonly untested, underlie researchers' interpretations about bins of utterances or factors/components generated by a quantitative method. An utterance or survey item is assigned to one and only one bin, subscale or factor/component when the researcher's analysis signals it can belong to multiple bins (Fryer & Nakao, 2020;Rogiers et al, 2020). This practice simplifies analyses but may well oversimplify what respondents mean.…”

Measurement and the Study of Motivation and Strategy Use: Determining If and When Self-report Measures are Appropriate

“…Whereas theoretical descriptions of a construct may be abstract and difficult to pin down, an operational definition is the specific way in which the construct is measured, including the exact prompts to which participants respond and the ways that data is collected. Consequently, if one wants to know what is meant by theoretical terms such as identity formation (Vriesema & McCaslin, 2020) or interest (Fryer & Nakao, 2020), one need only look to how those constructs are operationalized. In this regard, establishing this aspect of construct validity requires three elements (1) a clear articulation of the theoretical conceptualization, (2) a clear articulation around the measurement methodology, and (3) a coherent mapping between the conceptualization and the methodology.…”

“…A partial response to this challenge can be obtained by pointing back to the constructs of interest. Specifically, when the construct of interest is a learner's perception of intra-psychic states (e.g., beliefs, motivations), then it is sensible to suggest that the best way to uncover these perceptions is to ask the learner (Fryer & Nakao, 2020). In addition to this argument, however, articles in this special issue lay out an even more convincing reason for using self-report measures.…”

“…The articles in this issue represent different ways of answering that call. Articles by Fryer and Nakao (2020), Iaconelli and Wolters (2020), and Chauliac et al (2020) for example, adopt a measurement approach and focus on factors that can influence the reliability and validity of self-report data. Other articles; namely those by Durik and Jenkins (2020) and Moeller et al (2020); explore methods to enhance the evidentiary value of self-report data.…”