A Large-scale Evaluation of a Rubric for the Automatic Assessment of Algorithms and Programming Concepts

Alves, Nathalia da Cruz; Wangenheim, Christiane Gresse von; Hauck, Jean Carlo Rossa; Borgatto, Adriano Ferreti

doi:10.1145/3328778.3366840

Cited by 19 publications

(29 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…And, observing the availability of such assessments, even in automated ways for "traditional" block-based programming languages (e.g., Dr. Scratch (Moreno-León & Robles, 2015) or CodeMaster (Alves et al, 2020;Gresse von Wangenheim, 2018;Solecki et al, 2020), the need for such support also in the context of ML education becomes evident. We also observed that most tools do not provide any kind of support for teachers to monitor their students' learning.…”

Section: Discussionmentioning

confidence: 99%

Visual tools for teaching machine learning in K-12: A ten-year systematic mapping

et al. 2021

View full text Add to dashboard Cite

Teaching Machine Learning in school helps students to be better prepared for a society rapidly changing due to the impact of Artificial Intelligence. This requires age-appropriate tools that allow students to develop a comprehensive understanding of Machine Learning in order to become creators of smart solutions. Following the trend of visual languages for introducing algorithms and programming in K-12, we present a ten-year systematic mapping of emerging visual tools that support the teaching of Machine Learning at this educational stage and analyze the tools concerning their educational characteristics, support for the development of ML models as well as their deployment and how the tools have been developed and evaluated. As a result, we encountered 16 tools targeting students mostly as part of short duration extracurricular activities. Tools mainly support the interactive development of ML models for image recognition tasks using supervised learning covering basic steps of the ML process. Being integrated into popular block-based programming languages (primarily Scratch and App Inventor), they also support the deployment of the created ML models as part of games or mobile applications. Findings indicate that the tools can effectively leverage students’ understanding of Machine Learning, however, further studies regarding the design of the tools concerning educational aspects are required to better guide their effective adoption in schools and their enhancement to support the learning process more comprehensively.

show abstract

Section: Discussionmentioning

confidence: 99%

Visual tools for teaching machine learning in K-12: A ten-year systematic mapping

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Concerning construct validity, there also exists an indication of convergent validity based on the results of a correlation and factor analysis. These results indicate that the rubric can be used for a valid assessment of algorithm and programming concepts of App Inventor programs as part of a comprehensive assessment completed by other assessment methods, such as observations [49]. The assessment using the CodeMaster rubric is automated by performing a static code analysis.…”

Section: Screensmentioning

confidence: 95%

“…Adopting the Goal Question Metric approach [58], the objective of this study is defined as to analyze the 'demonstrated difficulty' of algorithms & programming concepts of App Inventor projects based on the CodeMaster rubric [49]. Here the term 'demonstrated difficulty' is defined as the volition, incentive, and opportunity to apply programming concepts in an App Inventor project shared via App Inventor Gallery, on which no further background information on the authors is provided.…”

Section: Methodsmentioning

confidence: 99%

“…The CodeMaster rubric can be regarded as reliable (Cronbach's alpha α=0.84) [49]. Concerning construct validity, there also exists an indication of convergent validity based on the results of a correlation and factor analysis.…”

Section: Screensmentioning

confidence: 99%

“…Therefore, the objective of this study is to analyze the 'demonstrated difficulty' in App Inventor projects. Adopting IRT, we analyze algorithms & programming items based on the CodeMaster rubric [49][50] by extracting them automatically from the code of App Inventor projects. The results provide information about the 'demonstrated difficulty' of the concepts application and their distribution among the App Inventor projects.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An Item Response Theory Analysis of Algorithms and Programming Concepts in App Inventor Projects

Alves

Wangenheim

Hauck

et al. 2021

Anais Do I Simpósio Brasileiro De Educação Em Computação (EDUCOMP 2021)

Self Cite

View full text Add to dashboard Cite

Computing education is often introduced in K-12 focusing on algorithms and programming concepts using block-based programming environments, such as App Inventor. Yet, learning programming is a complex process and novices struggle with several difficulties. Thus, to be effective, instructional units need to be designed regarding not only the content but also its sequencing taking into consideration difficulties related to the concepts and the idiosyncrasies of programming environments. Such systematic sequencing can be based on large-scale project analyses by regarding the volition, incentive, and opportunity of students to apply the relevant program constructs as latent psychometric constructs using Item Response Theory to obtain quantitative ?difficulty? estimates for each concept. Therefore, this article presents the results of a large-scale data-driven analysis of the demonstrated use in practice of algorithms and programming concepts in App Inventor. Based on a dataset of more than 88,000 App Inventor projects assessed automatically with the ANON rubric, we perform an analysis using Item Response Theory. The results demonstrate that the easiness of some concepts can be explained by their inherent characteristics, but also due to the characteristics of App Inventor as a programming environment. These results can help teachers, instructional and curriculum designers in the sequencing, scaffolding and assessment design of programming education in K-12.

show abstract

DrPython–WEB: A Tool to Help Teaching Well-Written Python Programs

Battistini

Isaia²,

Sterbini³

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

A Large-scale Evaluation of a Rubric for the Automatic Assessment of Algorithms and Programming Concepts

Cited by 19 publications

References 20 publications

Visual tools for teaching machine learning in K-12: A ten-year systematic mapping

Visual tools for teaching machine learning in K-12: A ten-year systematic mapping

An Item Response Theory Analysis of Algorithms and Programming Concepts in App Inventor Projects

DrPython–WEB: A Tool to Help Teaching Well-Written Python Programs

Contact Info

Product

Resources

About