Evaluating the Consequential Validity of the Research-Based Early Mathematics Assessment

Dong, Yixiao; Dumas, Denis; Clements, Douglas H.; Day-Hess, Crystal; Sarama, Julie

doi:10.1177/07342829231165812

Cited by 2 publications

(1 citation statement)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using the classic Bonferroni (Dunn, 1961) method, the new critical p value would be .0006. A more practical and equity-minded approach that has been used fruitfully (Dong et al, 2023), is to declare a critical p value of .01 for all DIF tests, with the understanding that some false positives may arise, but false positives are more desirable than false negatives.…”

Section: Phase 5—internal and External Validation Of Motesmentioning

confidence: 99%

Measuring original thinking in elementary school: Development and validation of a computational psychometric approach.

Acar,

Dumas,

Organisciak

et al. 2024

Journal of Educational Psychology

Self Cite

View full text Add to dashboard Cite

Creativity is highly valued in both education and the workforce, but assessing and developing creativity can be difficult without psychometrically robust and affordable tools. The open-ended nature of creativity assessments has made them difficult to score, expensive, often imprecise, and therefore impractical for school-or district-wide use. To address this challenge, we developed and validated the Measure of Original Thinking for Elementary School (MOTES) in five phases, including the development of the item pool and test instructions, expert validation, cognitive pilots, and validation of the automated scoring and latent test structure. MOTES consists of three game-like computerized activities (uses, examples, and sentences subscales), with eight items in each for a total of 24 items. Using large language modeling techniques, MOTES is scored for originality by our open-access artificial intelligence platform with a high level of agreement with independent subjective human ratings across all three subscales at the response level (rs = .79, .91, and .85 for uses, examples, and sentences, respectively). Confirmatory factor analyses showed a good fit with three factors corresponding to each game, subsumed under a higher-order originality factor. Internal consistency reliability was strong for both the subscales (H = 0.82, 0.85, and 0.88 for uses, examples, and sentences, respectively) and the higher-order originality factor (H = 0.89). MOTES scores showed moderate positive correlations with external creative performance indicators as well as academic achievement. The implications of these findings are discussed in relation to the challenges of assessing creativity in schools and research. Educational Impact and Implications StatementThis study introduces Measure of Original Thinking for Elementary School (MOTES), a validated assessment tool consisting of three game-like computerized activities, to address the challenge of assessing and nurturing creativity in education. By employing large language modeling methods, MOTES demonstrates remarkable correlations with human ratings, representing a significant leap forward in computational psychometrics for creativity assessment. The automated scoring methods employed by MOTES overcome cost-ineffectiveness and lack of affordability issues associated with traditional creativity tests, offering a reliable and affordable means of measuring original thinking in the verbal domain. These scores can be utilized alongside other indicators of school learning and academic achievement, highlighting the potential for creativity assessment to play a pivotal role in schoolbased programming and development efforts.

show abstract

Section: Phase 5—internal and External Validation Of Motesmentioning

confidence: 99%

Measuring original thinking in elementary school: Development and validation of a computational psychometric approach.

Acar,

Dumas,

Organisciak

et al. 2024

Journal of Educational Psychology

Self Cite

View full text Add to dashboard Cite

show abstract

Establishing benchmarks for assessing early mathematical competence in children

Menacho,

Aragón,

Arrigoni

et al. 2024

Front. Educ.

View full text Add to dashboard Cite

The Early Numeracy Test-Revised (ENT-r) development was followed by several provisional standardization processes to adapt to this new version. Subsequently, the ENT-r underwent translation into Spanish and a shift from a paper-and-pencil format to a computerized version, intending to make it accessible online for schools. This paper introduces the adapted Spanish version of ENT-r and outlines the provisional standardization procedure conducted with a group of Spanish children. In this initial pilot study, 141 children aged between 4 and 7 underwent individual assessments. Among them, 71 were girls (50.3%), and 70 were boys (49.6%). Selected from three public schools in a middle-class area, the children were evaluated by experienced researchers with expertise in assessing young children. The study involved three provisional statistical analyses using ENT-r data. Initially, a descriptive analysis was conducted, followed by a cross-age score comparison to assess scores across different age groups. Finally, a reliability study was performed. Preliminary results from these analyses indicate that the ENT-r demonstrates reliability, and its items effectively discriminate between prerequisite and counting tasks. Finally, an approximate statistical estimation was carried out regarding the level of mathematical competence, which is one of the parameters provided by the test, allowing the development of alternative improvement programs for the less prominent values.

show abstract

Evaluating the Consequential Validity of the Research-Based Early Mathematics Assessment

Cited by 2 publications

References 27 publications

Measuring original thinking in elementary school: Development and validation of a computational psychometric approach.

Measuring original thinking in elementary school: Development and validation of a computational psychometric approach.

Establishing benchmarks for assessing early mathematical competence in children

Contact Info

Product

Resources

About