Measuring socio-emotional skills in schools: simple questionnaires outperform behavioral tasks

Abstract

Recent empirical research has shown that improving non-cognitive skills such as grit, conscientiousness and self-control leads to higher academic achievement and better life outcomes. These findings have motivated the creation of educational programs targeting non-cognitive skills around the world. However, both theoretical and empirical works have raised concerns about the reliability of the different methods used to measure non-cognitive skills. The purpose of this study is to compare three methods of measurement of non-cognitive skills. To do so we capitalized on a unique data set collected across 97 middle schools in France, including a student-reported questionnaire, a teacher-reported questionnaire, and a behavioral task. Before analyzing the data, we polled 114 experts in cognitive development and education economics; most experts in both fields predicted that the behavioral task would be the best method. We found instead that the teacher questionnaire was more predictive of students' behavioral outcomes and of their grade progression, while the behavioral task was the least predictive. The three methods of measurement showed similar long-term stability and similar reliability, as measured by Cronbach's alpha. In addition, we show that the behavioral task is the least cost effective method to measure non cognitive skills. Our study suggests that behavioral tasks should not automatically be considered as the best method to measure non-cognitive skills.

The Role of Mindset in Education : A Large-Scale Field Experiment in Disadvantaged Schools

“…Note that the initial student survey also included the Academic Diligence Task developed by Galla et al, 2014 as a behavioral measure of diligence. Our prior was that a behavioral task would do better than questionnaires to measure diligence, but it turned out that it was no more reliable, and less valid, than self-reported and teacher-reported measures of diligence, a result that we show in a companion paper (Boon-Falleur et al, 2020). Given that the task is a costly instrument both financially and logistically, we removed it from the survey in the last two years and do not use it in this paper.…”

A variance decomposition of the Continuous Assessment of Interpersonal Dynamics.

“…This constrains the applicability of CAID for clinical assessment and research. A particular concern is the degree to which inferences can be made about people or dyads, in general, from data that were collected in one specific situation (e.g., Boon-Falleur et al, 2020;Hedge et al, 2018;Woods & Wright, 2019). In this study, we use Generalizability Theory (Cronbach et al, 1963) analyses to decompose the sources of variance in CAID ratings.…”

Childhood environmental adversity is not linked to lower levels of cooperative behaviour in economic games

“…Such a discrepancy between economic games and questionnaires is not an isolated finding and raises interesting questions about the validity of economic games to study realworld cooperation (Becker et al, 2012;Boon-Falleur et al, 2020;Chuang & Schechter, 2015;Dang et al, 2020;Lönnqvist et al, 2015). For instance, a recent systematic review and metaanalysis (Galizzi & Navarro-Martinez, 2019) found only weak evidence of a correlation (r = 0.14) between economic games and social behaviour in the field.…”