Assessment of computer and information literacy in ICILS 2013: Do different item types measure the same construct?

Abstract

The combination of different item formats is found quite often in large scale assessments, and analyses on the dimensionality often indicate multi-dimensionality of tests regarding the task format. In ICILS 2013, three different item types (information-based response tasks, simulation tasks, and authoring tasks) were used to measure computer and information literacy in order to balance technological and information-related aspects of computer and information literacy. The item types differ in the cognitive processes and the type of knowledge they measure and in the strands and aspects of the ICILS 2013 framework they address. In this article, we explored which factor models that assume item type factors or type of knowledge factors fit the data. For the factors of the best fitting models, regression analyses on SES, frequency of computer use, self-efficacy, and gender were computed to work out the different meanings and the convergent and discriminant validity of the factors. The results show that three-dimensional models with correlated factors for item type or type of knowledge fit best. Regression analyses discover substantive implications of between-item and within-item models. The effects are discussed and an outlook is given.

Exploring the structure of digital literacy competence assessed using authentic software applications

“…Instead, those dimensions seem to be defined by the software applications that students used in the assessment; they capture commonality among students' performance that seems to be due to their familiarity with the assessment tools and/or the context (RQ2). These results extend recent research that indicate item types may constitute measurable DL dimensions (Ihme et al 2017), suggesting that the measurement of DL performance may need to take account of the specific software applications used in the assessment. The existing theoretical frameworks on the dimensionality of DL may need revision, as most have assumed that DL is a generic competence, similar to reading literacy and numeracy.…”

“…How far DL as assessed through virtual environments specifically developed for the purpose of assessment reflects a person's competence when interacting with authentic software available in the commercial market remains unclear. Yet recent evidence, such as moderate correlations among DL performance shown with different commercial software tools in a longitudinal study (Lazonder et al 2020) and the task dependence identified among ICILS items (Ihme et al 2017), challenge the assumptions of task and tool independence of DL.…”

“…In a nuanced analysis, Ihme et al (2017) provided evidence that challenges both the conceptual frameworks and the unidimensional measurements of DL. Their analysis examined the relations among the assessment items used in ICILS 2013 to investigate whether these items form one unidimensional DL construct or need to be scaled as a multidimensional construct reflecting several DL dimensions.…”

“…It has to be noted that in studies suggesting a multidimensional DL structure, the dimensions were highly correlated (Ihme et al 2017;Siddiq et al 2017;Wilson et al 2017) such that reporting a unidimensional DL scale would be justified. However, these findings also alerted us of the need to be more cautious in adopting unidimensional 1 3 (Wilson et al 2018).…”

“…However, these findings also alerted us of the need to be more cautious in adopting unidimensional 1 3 (Wilson et al 2018). Further explorations of other DL assessments are also required to examine the dimensionality of DL along aspects other than the conceptual frameworks (Bundsgaard 2019;Ihme et al 2017).…”