Reliability of measuring constructs in applied linguistics research: a comparative study of domestic and international graduate theses

Abstract

The credibility of conclusions arrived at in quantitative research depends, to a large extent, on the quality of data collection instruments used to quantify language and non-language constructs. Despite this, research into data collection instruments used in Applied Linguistics and particularly in the thesis genre remains limited. This study examined the reported reliability of 211 quantitative instruments used in two samples of domestic and international theses in Applied Linguistics. The following qualities in measuring instruments were used to code the data: the instrument origin, instrument reliability, reliability facets examined, reliability computation procedures utilized, and the source of reliability reported (i.e., primary or cited). It was found that information about instrument origin was provided in the majority of cases. However, for 93 instruments, no reliability index was reported and this held true for the measurement of both language and non-language constructs. Further, the most frequently examined facet of reliability was internal consistency estimated via Cronbach’s alpha. In most cases, primary reliability for the actual data was reported. Finally, reliability was more frequently reported in the domestic corpus than in the international corpus. Findings are discussed in light of discursive and sociomaterial considerations and a few implications are suggested.

The Effectiveness of the Comparative Study of Languages with Different Structures in the Field of Phonetics (English and Kazakh Languages)

“…The formulation component processes the planning product and assembles it into a language form. This includes separate lexical, grammatical, and phonological operations that seem to be relatively independent, but the exact relationship of which has not been fully elucidated (Razavipour and Raji, 2022).…”

Interprofessional nursing education and the role of swift trust and task conflict in team creativity: the mediating role of team interactive behaviors

“…On the other hand, validity examines the accuracy of measurement, ensuring that the tests effectively measure the intended constructs (Middleton, 2022). Both reliability and validity are fundamental aspects of quantitative analytical approaches (Razavipour and Raji, 2022).…”

Scale quality in second-language anxiety and WTC: A methodological synthesis

“…Cronbach’s alpha is arguably the most common index of internal-consistency reliability in L2 research (as suggested by Plonsky & Derrick, 2016; Razavipour & Raji, 2022). Notably, Cronbach’s alpha remains popular despite a number of rigid assumptions that should be tenable for a reliability estimate based on alpha to be trustworthy (i.e., assumptions of unidimensionality, tau-equivalence, uncorrelated errors, and normal distribution of continuous variables); if these assumptions are not upheld, the estimate will not be accurate (as maintained by McNeish, 2018).…”