Analyzing student response processes to refine and validate a competency model and competency-based assessment task types

Abstract

Regarding competency-oriented teaching in higher education, lecturers face the challenge of employing aligned task material to develop the intended competencies. What is lacking in many disciplines are well-founded guidelines on what competencies to develop and what tasks to use to purposefully promote and assess competency development. Our work aims to create an empirically validated framework for competency-oriented assessment in the area of graphical modeling in computer science. This article reports on the use of the think-aloud method to validate a competency model and a competency-oriented task classification. For this purpose, the response processes of 15 students during the processing of different task types were evaluated with qualitative content analysis. The analysis shed light on the construct of graphical modeling competency and the cognitive demand of the task types. Evidence was found for the content and substantive aspect of construct validity but also for the need to refine the competency model and task classification.