This study aims to analyze and decompose the gap between White and non-White students’ test scores observed in the Brazilian National Evaluation System of Basic Education (SAEB)—2015. To do so, proficiency equations were estimated for each student race group (White, Brown, Black, and Indigenous) using the recentered influence function method, which generalizes the Oaxaca–Blinder decomposition to any point in the grade distribution. The proficiency of White students is higher than the other groups of students, and this difference increases throughout the distribution. Black students exhibit the lowest average test score, behind the Indigenous and then the Brown students. At the lower and middle levels of the math test score distribution, the racial gap is mainly due to the characteristics effect, which represents the part of the proficiency differential that is explained by the differences in observable characteristics of students. However, the gap at the upper level of grade distribution is mainly due the structural effect, which results from non-observable issues, such as discrimination and stereotype threats.