Why a Data‐Based Decision‐Making Intervention Works in Some Schools and Not in Others

Abstract

The use of data for adaptive, tailor-made education can be beneficial for students with learning difficulties. While evaluating the effects of a data-based decision-making (DBDM) intervention on student outcomes, considerable variation between intervention effects, ranging from highintervention effects to small or even negative intervention effects, across schools was found.The main purpose of this study was to investigate whether educator and school organizational characteristics are related to the effects of a DBDM intervention on student achievement growth by comparing 10 primary schools with strong intervention effects with 10 primary schools with no intervention effects on student achievement. Supportive and hindering factors were studied by means of surveys and interviews with school management teams, and by examining school reports from the project trainers. Results indicate that schools with strong intervention effects differed from schools with no intervention effects with regard to their teachers' teaching quality, staff's attitude toward DBDM, and the school data culture.

Evidence for the Feedback Role of Performance Measurement Systems

“…The PMSs that we study supply teachers with daily feedback on student learning. They support trend analysis and other meaningful comparisons, and are consistent with models of effective feedback and decision support (e.g., Keuning, Van Geel andVisscher 2017, Stecker, Fuchs andFuchs 2005). Both PMSs use data visualization (Cardinaels and Van Veen-Dirks 2010) to direct teachers' attention to students' shortcomings against learning objectives, to cue the need for remediation, and to support lesson planning to execute the remediation strategy.…”

“…The PMSs that we study supply teachers with daily feedback on student learning achievements and support trend analysis and other meaningful comparisons in both numeric and highly intuitive visual formats. 3 Their design and implementation are consistent with models of effective feedback and decision support (e.g., Keuning et al 2017, Stecker et al 2005. Teachers received focused training in using the systems and have a local "data driver" onsite to support their analyses.…”

“…4 Thus it may seem that the hypothesized effect is assured. However, even well-designed PMSs may fail to yield material improvements in learning outcomes if 1) PMS use comes at the expense of more effective teaching activities, 2) PMS use produces results that are weakly related to learning but support other valued outcomes (e.g., prosocial behavior), or 3) the PMSs confirm what the teacher already knows, providing no incremental information (Keuning et al 2017). Field interviews indicate that these threats are present at the research site.…”

“…Although the PMSs were adopted by the organization with the expectation of positive effects and their use by teachers is voluntary, a positive and economically meaningful feedback effect of teachers' use of the PMSs on student learning is far from assured. PMS use may fail to yield appreciable improvement in learning outcomes if: 1) PMS use comes at the expense of more effective teaching activities, 2) PMS use is intended to support other valued outcomes (e.g., prosocial behavior) that may be weakly related to learning, or 3) the PMSs confirm what the teacher already knows, providing no incremental information (Keuning et al 2017). Moreover, as Hattie (2009) documents in a comprehensive review of hundreds of meta-analyses of thousands of published intervention studies, PMSs, like many education interventions, may exhibit a statistically significant positive association with learning outcomes but at an effect size that is too small to warrant policymakers' attention.…”

Reflections on Teachers’ Data‐Based Decision Making

“…Differences between teachers and student teachers in Germany were examined in another study using a newly developed test of graph literacy that focused on student progress information (Zeuch, Forster, & Souvignier, ). An additional study in the Netherlands explored potential factors that may contribute to school differences in successful implementation of data‐based decision making (Keuning, van Geel, & Visscher, ).…”