Multiple imputation in big identifiable data for educational research: An example from the Brazilian education assessment system

Abstract

Almost all quantitative studies in educational assessment, evaluation and educational research are based on incomplete data sets, which have been a problem for years without a single solution. The use of big identifiable data poses new challenges in dealing with missing values. In the first part of this paper, we present the state-of-art of the topic in the Brazilian education scientific literature, and how researchers have dealt with missing data since the turn of the century. Next, we use open access software to analyze real-world data, the 2017 Prova Brasil , for several federation units to document how the naïve assumption of missing completely at random may substantially affect statistical conclusions, researcher interpretations, and subsequent implications for policy and practice. We conclude with straightforward suggestions for any education researcher on applying R routines to conduct the hypotheses test of missing completely at random and, if the null hypothesis is rejected, then how to implement the multiple imputation, which appears to be one of the most appropriate methods for handling missing data.

Progress Test: medical student absenteeism is suggestive of poor academic performance

“…At the institutional level, if the absence from the PT occurs in a systematic way, consisting mostly of a group with lower academic performance, then, with the increase in the proportion of absentees, the average performance in the PT of those present at the test tends to "apparently" increase the school result, making the latter biased 6,13,15,34 , as shown in this study. There is no certainty, but researchers working with missing data statistics suggest in systematic absence that, when the proportion of missing students is greater than 5 to 10% of students in a course, the results will be biased (in this study it was 21.6%); when, however, the absences occur randomly, they do not significantly affect the results 6,20 .…”

“…Then, linear regression was used to impute probable grades of absent students in the PT, based on 10 imputations by the monotonic method, using the PT grade as the dependent variable and phase in the course, EI, AAI and PAI as the independent variables. This method, known as multiple imputation of missing data 12,14,15,17,24 , assumes the model known as "missing at random (MAR)" for the outcome, that is, in the context of the present research, it is assumed that the PT scores of absent students were associated with the same independent variables as those who participated in the PT. With the averages of the PT of the students present at the test and the probable averages of the absent students obtained by imputation, the differences in the 11 phases were calculated and these averages were compared with the t test.…”

“…The occurrence of missing data, shown in clinical or educational research, reduces the sample size and variability, decreasing the statistical power and external validity of the studies 15,24 . On the contrary, the use of the multiple data imputation technique to calculate the absent students' probable grade in the PT and the inclusion of these results…”

“…There is also little concern about how to deal with missing data, even though there are statistical techniques, with robust literature, that can be used to impute presupposed values to the group of students that were absent from the PT [12][13][14] . Some techniques for imputing data to absentees, using the average of the ones present at the test, are not very concise 4 , but others, using multiple regression, are more precise 4,[14][15][16] .…”

“…These techniques are widely used in planning, research and evaluation of educational performance 4,6,14,15,17,18 . Students' grades and academic performance indicators can be used to verify whether there is any systematic difference between being present and being absent from the PT.…”