How Hard Will this Task Be? Developments in Analyzing and Predicting Question Difficulty in the Bebras Challenge

Abstract

Predicting the difficulty level of a task on the concepts of computer science or computational thinking, like in the Bebras Challenge, proves to be really hard. Question difficulty breaks down in content difficulty, stimulus difficulty and task difficulty. Several instruments are suggested to predict the overall difficulty level, like using a questionnaire or a rubric; these instruments are applied on the data of a recent contest and proved useful. Relative scoring could also turnout helpful. Especially on content difficulty easy applicable solutions are lacking.

Development and Validation of the Computational Thinking Test for Elementary School Students (CTT-ES): Correlate CT Competency With CT Disposition

“…Hubwieser and Mühling (2015) used the IRT method to examine a more dated version (i.e., the German Bebras Contest of 2009) than the versions referenced in the current study. Other studies have used qualitative evaluation (e.g., content analysis or rubrics) or quantitative measures, such as questionnaires or success rate for identifying item difficulty (Izu et al, 2017; van der Vegt, 2018). Past studies have pointed out that the chosen items from the Bebras item pool represent a joint construct; nevertheless, there exist problems with the quality of items (Hubwieser & Mühling, 2015).…”