What to Make Of and How to Interpret Process Data

Goldhammer, Frank; Zehner, Fabian

doi:10.1080/15366367.2017.1411651

Cited by 23 publications

(28 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Comparably, the indicator of general sourcing revealed a mixture of these patterns. Generally, the results support the overall postulation that data from log‐files can be filtered and integrated in a theory‐driven way to meaningfully represent individuals’ strategies or skills (Goldhammer & Zehner, ). However, they also show that the interpretation of process variables strongly depends on the framework which conceptualizes a construct of interest.…”

Section: Discussionsupporting

confidence: 71%

“…This study therefore uses a standardized assessment of multiple document comprehension (MDC) to evaluate whether meaningful episodes (‘states’; Kroehne & Goldhammer, ) of sourcing occur as expected by current theories on the functions of sourcing. These episodes are examined for their relations to characteristics of persons, documents, tasks, and the administration of the MDC assessment, to validate their intended meaning and reduce ambiguities in their interpretation (Goldhammer & Zehner, ).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Validating process variables of sourcing in an assessment of multiple document comprehension

Hahnel

Kroehne

Goldhammer

et al. 2019

Brit J of Edu Psychol

Self Cite

View full text Add to dashboard Cite

Background With digital technologies, competence assessments can provide process data, such as mouse clicks with corresponding timestamps, as additional information about the skills and strategies of test takers. However, in order to use variables generated from process data sensibly for educational purposes, their interpretation needs to be validated with regard to their intended meaning. Aims This study seeks to demonstrate how process data from an assessment of multiple document comprehension can be used to represent sourcing, which summarizes activities for the consideration of the origin and intention of documents. The investigated process variables were created according to theoretical assumptions about sourcing, and systematically tested for differences between persons, units (i.e., documents and items), and properties of the test administration. Sample The sample included 310 German university students (79.4% female), enrolled in several bachelor's or master's programmes of the social sciences and humanities. Methods Regarding the hierarchical data structure, the hypotheses were analysed with generalized linear mixed models (GLMM). Results The results mostly revealed expected differences between individuals and units. However, unexpected effects of the administered order of units and documents were detected. Conclusions The study demonstrates the theory‐informed construction of process variables from log‐files and an approach for empirical validation of their interpretation. The results suggest that students apply sourcing for different reasons, but also stress the need of further validation studies and refinements in the operationalization of the indicators investigated.

show abstract

Section: Discussionsupporting

confidence: 71%

Section: Introductionmentioning

confidence: 99%

Validating process variables of sourcing in an assessment of multiple document comprehension

Hahnel

Kroehne

Goldhammer

et al. 2019

Brit J of Edu Psychol

Self Cite

View full text Add to dashboard Cite

show abstract

“…In doing so, the assessment of inquiry must be developed to properly capture students' interactions with inquiry tasks, such as their patterns of actions, sequences, and strategic behavior, as observable evidence for the inquiry process. Both theoretical and empirical evidence are needed to ensure that the reasoning from the process indicators (e.g., sequence of actions, number of trials) to the specific latent process (e.g., multivariable reasoning) is justifiable (Goldhammer & Zehner, 2017).…”

Section: Assessing Inquiry As An Instructional Outcomementioning

confidence: 99%

More isn't always better: The curvilinear relationship between inquiry-based teaching and student achievement in science

Teig

Scherer

Nilsen

2018

Learning and Instruction

100

View full text Add to dashboard Cite

Considerable research has demonstrated that teachers' self-efficacy plays a major role in implementing instructional practices. Only few studies, however, have examined the interplay between how teachers' self-efficacy and the challenges that lie outside their influence are related to their implementation of cognitive-activation strategies (CASs), especially in science classrooms. Using the Trends in Mathematics and Science Study 2015 data from science teachers in Grades 4, 5, 8, and 9, we explored the extent to which teachers' self-efficacy in science teaching and the perceived time constraints explained variations in the enactment of general and inquiry-based CAS. Findings from the overall sample showed that highly self-efficacious teachers reported more frequent implementation of both general and inquiry-based CAS, whereas those who perceived strong time constraints reported a less frequent use of inquiry-based CAS. These relationships also existed across grade levels, except on the relations between perceived time constraint and inquiry-based CAS, which was only significant for the science teachers in Grade 9. We discuss these findings in light of variations in the core competencies of science curriculum, teachers' competences, and the resources for science activities between primary and secondary education. We also point to the theoretical implications of this study for enhancing the conceptual understanding of generic and specific aspects of CAS and the practical implications for teacher education, professional development, and educational policy.

show abstract

“…Which states are relevant and how the desired indicators can be meaningfully extracted from the sequence of states must be defined in an assessment framework (e.g., Goldhammer and Zehner 2017;Mislevy et al 2012;OECD 2016), going beyond simple taxonomy of paradata with prototypical examples. Such a framework elaborates the targeted (latent) construct (e.g., cognitive states of information processing), identifies observable evidence for that construct and the item content needed to elicit the desired behavior captured by considered states.…”

Section: Statesmentioning

confidence: 99%

How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items

Kroehne

Goldhammer

2018

Behaviormetrika

Self Cite

View full text Add to dashboard Cite

Log data from educational assessments attract more and more attention and largescale assessment programs have started providing log data as scientific use files. Such data generated as a by-product of computer-assisted data collection has been known as paradata in survey research. In this paper, we integrate log data from educational assessments into a taxonomy of paradata. To provide a generic framework for the analysis of log data, finite state machines are suggested. Beyond its computational value, the specific benefit of using finite state machines is achieved by separating platform-specific log events from the definition of indicators by states. Specifically, states represent filtered log data given a theoretical process model, and therefore, encode the information of log files selectively. The approach is empirically illustrated using log data of the context questionnaires of the Programme for International Student Assessment (PISA). We extracted item-level response time components from questionnaire items that were administered as item batteries with multiple questions on one screen and related them to the item responses. Finally, the taxonomy and the finite state machine approach are discussed with respect to the definition of complete log data, the verification of log data and the reproducibility of log data analyses.

show abstract

What to Make Of and How to Interpret Process Data

Cited by 23 publications

References 23 publications

Validating process variables of sourcing in an assessment of multiple document comprehension

Validating process variables of sourcing in an assessment of multiple document comprehension

More isn't always better: The curvilinear relationship between inquiry-based teaching and student achievement in science

How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items

Contact Info

Product

Resources

About