Scale development: ten main limitations and recommendations to improve future research practices

Abstract

The scale development process is critical to building knowledge in human and social sciences. The present paper aimed (a) to provide a systematic review of the published literature regarding current practices of the scale development process, (b) to assess the main limitations reported by the authors in these processes, and (c) to provide a set of recommendations for best practices in future scale development research. Papers were selected in September 2015, with the search terms "scale development" and "limitations" from three databases: Scopus, PsycINFO, and Web of Science, with no time restriction. We evaluated 105 studies published between 1976 and 2015. The analysis considered the three basic steps in scale development: item generation, theoretical analysis, and psychometric analysis. The study identified ten main types of limitation in these practices reported in the literature: sample characteristic limitations, methodological limitations, psychometric limitations, qualitative research limitations, missing data, social desirability bias, item limitations, brevity of the scale, difficulty controlling all variables, and lack of manual instructions. Considering these results, various studies analyzed in this review clearly identified methodological weaknesses in the scale development process (e.g., smaller sample sizes in psychometric analysis), but only a few researchers recognized and recorded these limitations. We hope that a systematic knowledge of the difficulties usually reported in scale development will help future researchers to recognize their own limitations and especially to make the most appropriate choices among different conceptions and methodological strategies.

Translation and validation of the Mind-Wandering Test for Spanish adolescents

“…DeVellis (2003) suggested the combined use of EFA and CFA for more consistent results on the psychometric indices of new scales. Recently, this suggestion of considering the combined use of EFA and CFA during the evaluation of construct validity of new measures has been approved by other authors, in order to provide more consistent psychometric results (Morgado, Meireles, Neves, Amaral and Ferreira, 2017).…”

Measuring corporate social responsibility: A critical review of survey instruments

“…Convergent validity examines whether a scale's score is associated with the other variables and measures of the same construct to which it should be related (Morgado et al, 2017). Conceptually, failure to find correlations among measures naturally elevates concerns about the validity of the measurement (Cunningham, Preacher, & Banaji, 2001).…”

Are Filipino Students’ L2 Learning Goals Performance- or Mastery-oriented? An Explanatory Sequential Analysis

“…To measure the independent variable, the researchers developed the Goal Orientation in a L2 Scale (GOALS) following the procedures for psychometric test development (Morgado et al, 2017). A separate sample of 180 participants from the same University and enrolled in the same EAP course participated in the test development.…”