Applying the science of learning to EdTech evidence evaluations using the EdTech Evidence Evaluation Routine (EVER)

Consolidated Benchmark for Efficacy and Effectiveness Frameworks in EdTech

“…Several quality assessment criteria exist, including those by Gough (2017), which provide a three-level (high, medium, and low) framework of criteria for appraising the quality and relevance of evidence. The 'Edtech Evaluation Evidence Routine' (EVER) proposed by Kucirkova, Brod, and Gaab (2023) for some EdTech products/solutions, a randomised controlled trial (RCT) might be appropriate, for others, such as a screening tool, for example, a predictive validity study would be more suitable (Kucirkova, Brod & Gaab, 2023). EVER is described as to be applicable to all types of EdTech aiming to improve children's learning or to modify learners' behaviour, as well as those that integrate assessment and intervention.…”

“…Regarding the relevance of the evidence, ask: What is the relevance of the particular focus of the individual study for addressing its aims? To judge the answers to these questions in terms of high, medium, and low in the context of EdTech, the aforementioned five EVER criteria (methodological quality, outcomes strength, predictive value, generalizability, and ethics & transparency) based on Kucirkova, Brod, and Gaab (2023), were used.…”

“…This aspect was determined based on the criteria proposed by Singal, Higgins, and Waljee (2014), encompassing "study design, patient populations, intervention design, data analysis, and result reporting." Given a considerable conceptual overlap between these criteria and those proposed by Kucirkova, Brod, and Gaab (2023) for assessing the internal and external validity of EdTech studies, the frameworks scoring 3-5 on quantitative dimensions of the EVER routine were considered as meeting the criteria for rigorous efficacy studies.…”

“…A systematic literature search identified 65 frameworks that target the efficacy or effectiveness of K12 EdTech products. The frameworks were analysed in relation to their rigour, using the science of learning principles embedded in the EdTech Evidence Evaluation Routine (Kucirkova, Brod & Gaab, 2023). The results were synthesised into a consolidated benchmark that categorises the available frameworks at three levels based on the rigour applied to their assessments.…”

Advancing the understanding of children’s digital engagement: responsive methodologies and ethical considerations in psychological research

“…Without designing measures and analyses that reflect the digital experiences of historically marginalized communities, psychological science risks generating results that center the experiences of privileged families. Already, the science of learning, underscores the importance of employing diverse methods and carefully selecting the most appropriate method for a specific context, research question, and increasingly, digital environment ( Kucirkova et al, 2023 ). Being mindful of this diversity and choosing methodologies that best align with the research problem not only demonstrates methodological rigor but also fulfills an ethical obligation to address pressing questions regarding digital media.…”