Measurement Issues in Research on Shadow Education: Challenges and Pitfalls Encountered in TIMSS and PISA

Abstract

Expanding numbers of researchers are focusing on the scale and impact of private supplementary tutoring. Such tutoring is widely called shadow education, since much of its curriculum mimics that of regular schooling. Although shadow education has expanded significantly worldwide and is now recognized to have far-reaching significance, research faces methodological and conceptual challenges. This article focuses on analyses of shadow education data from the Third (or Trends in) International Mathematics and Science Study (TIMSS) and the Programme for International Student Assessment (PISA). An initial problem arises from definitions of shadow education and therefore from research focus. Further challenges arise from the initial phrasing and then translation of items in international questionnaires. The article notes that some studies have been grounded in problematic data, which has led to misleading pictures. Methods and approaches are maturing, but much refinement remains necessary for an adequate understanding of the nature and implications of shadow education.

A two‐level study of predictors of private tutoring attendance at the beginning of secondary schooling in Germany: The role of individual learning support in the classroom

“…It would be informative to repeat the same analyses for older students when learning content becomes more complex. The lack of income effects on tutoring attendance might also be due to the fact that we did not explicitly ask for differentiation between paid and free private tutoring (Bray & Kobakhidze, ). Additionally, our data did not make it possible to check whether wealthier parents chose more expensive and more high‐quality tutoring for their children.…”

“…As private tutoring costs are less relevant for wealthy families, tutoring attendance is theoretically and empirically dependent on families’ economic resources (Tansel & Bircan, ; Buchmann et al , ; Jung & Lee, ; Kim & Park, ; Guill, ; Hille et al , ). The effects of families’ economic resources might even be underestimated, as not all studies explicitly differentiate between paid and free private tutoring in their questionnaires (Bray & Kobakhidze, ). As predicted, there is evidence from qualitative (Kirss & Jokić, ) and quantitative analyses (Smyth, ; Guill, ) that private tutoring also increases in line with families’ educational aspirations and, as educational aspirations depend on social status (Boudon, ), with their social status (Smyth, ) or education level (Davies, ; Tansel & Bircan, ; Dang, ; Kim & Park, ).…”

Shadow educations: mapping the global discourse

“…International research collaboration is desirable for more concerted and systematic efforts to yield meaningful and conclusive findings. On the quantitative domain, developing rigorous instruments and research methods in the elaboration and use of international comparative data on private tutoring, as indicated by Bray and Kobakhidze (2014), is needed in order to arrive at an integrated vision of the phenomenon of private tutoring in different contexts worldwide. Meta-analyses may also yield powerful results, provided they are based on solid research studies.…”

“…In this respect, fine-tuning of international comparative data derived from the TIMSS and PISA (Programme for International Student Assessment) is imperative for more meaningful and coherent conclusions to be derived. The methodological issues raised by a few scholars on large-scale studies in shadow education are worth noting (Bray, 2010b;Bray & Kobakhidze, 2014). Other types of research, such as explanatory, experimental and intervention studies are also needed.…”

When One Door Closes: The Impact of the Hagwon Curfew on the Consumption of Private Tutoring in the Republic of Korea

“…We estimate mean and heterogeneous effects by educational level and socioeconomic status applying difference-in-differences (DD) estimators to the 2009-2012 waves of the SPEE. By doing so, we are able to overcome many of the information problems identified by Bray and Kobakhidze (2014) in previous studies of private tutoring.…”