Toward More Generalizable CS and CT Instruments

Rachmatullah, Arif; Vandenberg, Jessica; Wiebe, Eric N.

doi:10.1145/3502718.3524790

Cited by 6 publications

(7 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast, Indonesian students' attitudes and motivation to learn science and mathematics have been reported above average compared with other countries (Mullis et al, 2016; OECD, 2019). Several of our previous international studies have confirmed that Indonesian students have higher attitudes toward science, STEM, and CS compared with students in Korea and the United States (Rachmatullah et al, 2018, 2022; Shin et al, 2018). Such elevated STEM attitudes might be the result of the country's policy efforts toward establishing more stable social and financial conditions for its citizens through the development of high value STEM academic and career capital.…”

Section: Socio‐cultural Context Of Indonesiamentioning

confidence: 76%

Exploring middle school students' interests in computationally intensive science careers: The CISCI instrument validation and intervention

Rachmatullah

Wiebe

2022

Science Education

Self Cite

View full text Add to dashboard Cite

Computationally intensive science (CIS)‐related disciplines and careers are predicted to be among the cutting‐edge fields and high‐demand jobs. Such disciplines and careers demand expertise in both traditional scientific knowledge and computer science (CS)‐related understanding and skills. Preparing educational systems for developing interest and abilities for such career paths require a new set of research tools. In this study, we developed and validated a multidimensional instrument called computationally intensive science career interests (CISCI) to measure middle school students' interests in CIS careers. We also explored several predictors of students' interests in such careers and examined the impact of a 4‐day online synchronous scientific computational modeling activity on students' career interests. A total of 934 Indonesian middle school students (aged 11–14) participated in this study. A combination of the classical test theory and item response theory approaches was used to validate the CISCI instrument. Multiple linear regression tests were run to identify the predictors of students' career interests, and paired‐sample t‐tests were used to examine a scientific computational modeling activity's impact on students' career interests. The results revealed that CISCI was a psychometrically valid and reliable instrument to measure students' career interests. We also found that science and CS attitudes, computational thinking, and prior experience in CS‐related activities were significant predictors of students' career interests. In addition, computational modeling activity significantly influenced the frequency of students discussing such career paths with their parents. This study underscores the importance of engaging students in CS‐integrated science learning activities to help develop their interests in CIS jobs.

show abstract

Section: Socio‐cultural Context Of Indonesiamentioning

confidence: 76%

Exploring middle school students' interests in computationally intensive science careers: The CISCI instrument validation and intervention

Rachmatullah

Wiebe

2022

Science Education

Self Cite

View full text Add to dashboard Cite

show abstract

“…Differential Item Functioning (DIF) is statistical approach that is usually employed to determine whether there are biases in response patterns between groups for certain items (e.g. according to gender as done by Rachmatullah et al, 2022;Sovey et al, 2022 or countries as done by Rachmatullah et al, 2022).…”

Section: Differential Item Functioningmentioning

confidence: 99%

“…In the present context we employ DIF to determine whether i) there are differences in the response patterns between students in grades 3-4 and grades 5-6, and whether grade-specific IRT models should be employed instead of a single model, and ii) the instrument is fair with respect to gender as instruments such as these are often employed to determine whether gender gaps exist and whether interventions help address them given the lack of diversity in computing-related fields (Rachmatullah et al, 2022). While it would have been interesting to establish fairness in terms of socio-economic status, this type of information is considered sensitive in the region.…”

Section: Differential Item Functioningmentioning

confidence: 99%

“…It would thus be interesting to expand the validation to students in other countries to determine to what extent the results generalise or are influenced by local curricula. This would also provide the opportunity to conduct Differential Item Functioning across different countries to establish to what extent the cCTt is generalisable (Rachmatullah et al, 2022).…”

Section: Proposing Grade-specific Student Proficiency Profilesmentioning

confidence: 99%

See 1 more Smart Citation

competent Computational Thinking Test

El-Hamamsy¹,

Zapata-Cáceres²,

Barroso³

et al. 2022

PsycTESTS Dataset

View full text Add to dashboard Cite

The introduction of computing education into curricula worldwide requires multi-year assessments to evaluate the long-term impact on learning. However, no single Computational Thinking (CT) assessment spans primary school, and no group of CT assessments provides a means of transitioning between instruments. This study therefore investigated whether the competent CT test (cCTt) could evaluate learning reliably from grades 3 to 6 (ages 7-11) using data from 2709 students. The psychometric analysis employed Classical Test Theory, normalised z-scoring, Item Response Theory, including Differential Item Functioning and PISA's methodology to establish proficiency levels. The findings indicate that the cCTt is valid, reliable and gender-fair for grades 3-6, although more complex items would be beneficial for grades 5-6. Grade-specific proficiency levels are provided to help tailor interventions, with a normalised scoring system to compare students across and between grades, and help establish transitions between instruments. To improve the utility of CT assessments among researchers, educators and practitioners, the findings emphasise the importance of i) developing and validating gender-fair, grade-specific, instruments aligned with students' cognitive maturation, and providing ii) proficiency levels, and iii) equivalency scales to transition between assessments. To conclude, the study provides insight into the design of longitudinal developmentally appropriate assessments and interventions.

show abstract

“…Although access to developmentally appropriate CS & CT education can increase students' skills from a young age (Bers et al, 2014(Bers et al, , 2022aRelkin et al, 2021), several studies suggest that perception of the discipline can also influence performance. Rachmatullah et al (2022) found that the gender-performance gap was more prevalent in countries where the "socio-cultural context" tends to promote such stereotypes and "influenc [e] gender diversity in the CS field". This is corroborated by Hinckle et al (2020) who found that student learning was not directly influenced by prior experience, but was mediated by their perception of CS.…”

Section: The Influence Of Social and Structural Barriers On Learning-...mentioning

confidence: 99%

A case for co-construction with teachers in curricular reform: Introducing computer science in primary school

El-Hamamsy

Bruno

Kovacs

et al. 2022

Proceedings of the 24th Australasian Computing Education Conference

View full text Add to dashboard Cite

With the introduction of Computer Science (CS) into curricula worldwide, researchers have investigated whether CS could be introduced transversally, as a support other disciplines. Few however consider both student learning and the teachers' perspective in their assessments. In a co-constructive approach to translational research, we collaborated with teachers, in two case studies involving two classes each, to investigate how CS content could be used transversally. More specifically, teacher inputs and student learning data were combined in a mixed methods analysis to determine whether two CS Unplugged (CSU) activities from the curriculum could be leveraged to teach disciplinary content in primary school. The findings indicated that the CSU activities could be leveraged for maths and spelling, but require validation at a larger scale. More important, though, are the takeaways of the co-constructive experience with teachers. Interestingly the benefits of co-construction went both ways. While researchers gained a deeper understanding of the effectiveness of the activities, teachers i) appreciated having detailed student learning analyses, devoid of biases and impressions, ii) changed their perspective about researchers, and iii) were open to other collaborative research initiatives which they found to be drivers of change and innovation in their practice. The way co-construction was approached also impacted the teachers' perception of their experience. Indeed, while one study was researcher-driven and included teachers, the other was teacherdriven and included researchers. When teacher-driven, the teachers felt they had a more active role, thus reflecting more on the study design, results and implications. Provided the positive impact that co-construction may have on innovation in teacher practices, different co-construction modalities must be investigated, as well as their implications on all stakeholders, including researchers.

show abstract

Toward More Generalizable CS and CT Instruments

Cited by 6 publications

References 24 publications

Exploring middle school students' interests in computationally intensive science careers: The CISCI instrument validation and intervention

Exploring middle school students' interests in computationally intensive science careers: The CISCI instrument validation and intervention

competent Computational Thinking Test

A case for co-construction with teachers in curricular reform: Introducing computer science in primary school

Contact Info

Product

Resources

About