The handling of missing binary data in language research

Abstract

Researchers are frequently confronted with unanswered questions or items on their questionnaires and tests, due to factors such as item difficulty, lack of testing time, or participant distraction. This paper first presents results from a poll confirming previous claims (Rietveld & van Hout, 2006; Schafer & Gra- ham, 2002) that data replacement and deletion methods are common in research. Language researchers declared that when faced with missing answers of the yes/no type (that translate into zero or one in data tables), the three most common solutions they adopt are to exclude the participant’s data from the analyses, to leave the square empty, or to fill in with zero, as for an incorrect answer. This study then examines the impact on Cronbach’s α of five types of data insertion, using simulated and actual data with various numbers of participants and missing percentages. Our analyses indicate that the three most common methods we identified among language researchers are the ones with the greatest impact n Cronbach's α coefficients; in other words, they are the least desirable solutions to the missing data problem. On the basis of our results, we make recommendations for language researchers concerning the best way to deal with missing data. Given that none of the most common simple methods works properly, we suggest that the missing data be replaced either by the item’s mean or by the participants’ overall mean to provide a better, more accurate image of the instrument’s internal consistency.

Missing Data Management Practices in L2 Research: The Good, The Bad and The Ugly

“…Similarly, future studies might also focus on whether there is a change in the missing data analytic practices over time. Additionally, contrary to the present synthesis which adopted a slightly broad scope, future research might take a narrow focus in investigating the missing data problem (e.g., missingness issue in binary data; see Pichette et al, 2015). For instance, a methodological synthesis looking at the studies that employed surveys and questionnaires might tell us more about the state of the art of missing data in L2 research because surveys and questionnaires are notoriously known for their missing data rate.…”

“…However, alhtough there is a growing body of research on missing data, this has not unfortunately been reflected in the field of SLA. The only study that examined the missing data issue in language research is Pichette et al's (2015) review. Pichette et al investigated the missing binary data issue (i.e., missing responses to dichotomous items/questions such as yes/no questions or agree/disagree items) and what kinds of methods language researchers commonly employed to deal with such missing issues in binary data.…”

Impact on Cronbach's alpha of simple treatment methods for missing data

“…Finally, Pichette et al (2015) studied Cronbach's α and discovered that replacing a nonresponse by the mean for that item is a better alternative than simple methods like leaving the square empty, or filling in with zero, as for an incorrect answer. However, that study had certain limitations, such as not including advanced methods like multiple imputation.…”

“…Sharpe (2013) invoked resistance to statistical innovation to describe that phenomenon. Another argument refers to the fact that providing information about participant exclusion based on their missing data can raise too many questions and decrease the chance of being accepted in a peer-reviewed journal (Pichette, Béland, Jolani, & Lesniewska, 2015). There also remains the issue of the effect these nonresponses can have on the psychometric properties of tests.…”