Executive functioning rating scales: Ecologically valid or construct invalid?

Abstract

Objective: Executive functions are commonly measured using rating scales and performance tests. However, replicated evidence indicates weak/nonsignificant cross-method associations that suggest divergent rather than convergent validity. The current study is the first to investigate the relative concurrent and predictive validities of executive function tests and ratings using (a) multiple gold-standard performance tests, (b) multiple standardized rating scales completed by multiple informants, and (c) both performance-based and ratings-based assessment of academic achievement-a key functional outcome with strong theoretical links to executive function. Method: A well-characterized sample of 136 children oversampled for ADHD and other forms of child psychopathology associated with executive dysfunction (ages 8 -13; 68% Caucasian/non-Hispanic) completed a counterbalanced series of executive function and academic tests. Parents/teachers completed executive function ratings; teachers also rated children's academic performance. Results: The executive function tests/ratings association was modest (r ϭ .30) and significantly lower than the academic tests/ratings association (r ϭ .63). Relative to ratings, executive function tests showed significantly higher cross-method predictive validity and significantly better within-method prediction; executive function ratings failed to demonstrate improved within-method prediction. Both methods uniquely predicted academic tests and ratings. Conclusion: These findings replicate prior evidence that executive function tests and ratings cannot be used interchangeably as executive function measures in research and clinical applications, while suggesting that executive function tests may have superior validity for predicting academic behavior/achievement.

Die Rolle elternbeurteilter exekutiver Funktionen zur Schulleistungsprognose

“…Das Lehrkraftrating hatte hier den größten Vorhersagewert, ge-folgt von Elternbeurteilungen und Performanztests. In einer der wenigen Studien, in denen der inkrementelle prädiktive Wert der Elternbeurteilungen über Performanztests hinaus untersucht wurde, erfassten Dekker et al (2017) (Soto et al, 2020). Allerdings klärten die Performanztests einen höheren einzigartigen Varianzanteil auf als die Fremdbeurteilungen.…”

“…Ein Vorteil der vorliegenden Studie ist, dass mathematische und Leseleistungen separat betrachtet wurden, während in früheren Studien häufig Gesamtwerte genutzt wurden (Diamantopoulou, Rydell, Thorell & Bohlin, 2007;Langberg, Dvorsky & Evans, 2013;Soto et al, 2020;Wirth, Reinelt, Gawrilow & Rauch, 2015). Der gesamte aufgeklärte Varianzanteil war dabei für die Leseleistungen deut lich höher als für die Mathematikleistungen, wie es auch in früheren Studien der Fall war (Thorell, 2007;Ten Eycke & Dewey, 2016).…”

“…B. Gerst et al, 2017;Jacobson, Williford & Pianta, 2011;Roebers et al, 2014;Soto et al, 2020). Interessanterweise sind die Korrelationen der EF-Performanztests mit den Schulleistungstests höher als mit den Elternbeurteilungen, die dasselbe Konstrukt messen sollten.…”

“…Dennoch stellt sich in der Praxis die Frage der EF-Diagnostik häufig bei Kindern mit Auffälligkeiten, weswegen insbesondere die Einbeziehung dieser Populationen wichtig ist (vgl. Soto et al, 2020). So deuten z.…”

Executive function measurement in urban schools: Exploring the links between performance-based metrics and teacher ratings

“…Behavior rating scales became an effective way to gauge everyday executive functioning for younger populations, as they can provide a unique perspective on different aspects of executive functions, some of which may not be measured by performance-based tasks (e.g., Isquith et al, 2004;Soto et al 2020). In addition, they can be completed by parents or teachers because they have more extensive experience with the child.…”