Exploring Multiple Goals Balancing in Complex Problem Solving Based on Log Data

Abstract

Multiple goals balancing is an important but not yet fully validated dimension of complex problem solving (CPS). The present study used process data to explore how solvers clarify goals, set priorities, and balance conflicting goals. We extracted behavioral indicators of goal pursuit from the log data of 3,201 students on the third subtask of the “Ticket” task in the PISA 2012 CPS test. Cluster analysis was used to identify 10 groups that varied in goal pursuit behavior. Logistics and least-squares regression analysis were used to explore how goal pursuit affected task scores and CPS proficiency. The results showed that competent solvers clarified goals and weighed priorities more effectively. They also made trade-offs between conflicting goals. The importance of theoretically-driven log data analysis and coping strategies in the face of multiple goals conflict scenarios was discussed.

Logfile analyses of successful and unsuccessful strategy use in complex problem-solving: a cross-national comparison study

“…In the context of computer-based assessment, students' operations and interactions with the problem and test environment can be recorded, resulting in what we call logfiles (Zoanetti and Griffin 2017) and providing new opportunities to discover and conduct new forms of data analysis in educational assessment. In a CPS environment, logfile data have been used to explore students' problem-solving behaviours (Tóth et al 2017), exploration strategies Chen et al 2019;Greiff et al 2015c;Ren et al 2019), problemsolving proficiency and test-taking motivation (Zoanetti and Griffin 2017), and thus help us to gain a much deeper understanding of how they interact with problems and how they behave during the problem-solving and test-taking process. Using logfile analysis, we focused on their exploration behaviour in the problem-solving process.…”

A systematic review of empirical studies using log data from open‐ended learning environments to measure science and engineering practices

“…For example, Peffer et al (2019) used the k‐means algorithm to cluster students and explore the differences in their inquiry approaches. The third category, prediction (34 papers), includes studies that use the extracted features to predict an outcome variable, which can be categorical (classification) or numerical (regression). Studies in this category have adopted a variety of models and techniques, including regression models (McBride et al, 2016; Owen et al, 2016; Ren et al, 2019; Sawyer et al, 2018), decision trees (Doleck et al, 2014; Gobert et al, 2015; Malkiewich et al, 2016; Xing et al, 2021), random forests (Han et al, 2019; Qiao & Jiao, 2018) and neural networks (Cock et al, 2021; Käser & Schwartz, 2019; Poitras et al, 2019; Reilly & Dede, 2019). The fourth category, modelling (8 papers), denotes studies aimed at directly representing the underlying cognitive processes in the model.…”

“…From the 14 papers in this category, one used information criteria to assess the quality of the clustering and another assessed clustering stability (Käser & Schwartz, 2020). Five papers computed heuristics such as the Silhouette score (eg, Eichmann et al, 2020; Ren et al, 2019) to estimate separability of the different groups. Another frequently used approach (five papers) is to examine how well the resulting clusters separate from each other based on measures such as in‐task performance (Peffer et al, 2019; Zhang et al, 2017), student majors (Peffer et al, 2019), learning gain based on pre/post‐tests (Fratamico et al, 2017) and science grades and standardized assessment scores (Käser & Schwartz, 2020).…”

Moving Beyond Two Goals: An Integrative Review and Framework for the Study of Multiple Goals

“…For instance, a simultaneous pursuit can occur when people discover a multifinal means—a subgoal that can satisfy more than one goal at the same time (e.g., I will use a treadmill desk while working; Köpetz et al, 2011). The ability to find such subgoals relies partly on individuals’ ability to process complex information (Reichman et al, 2018; Ren et al, 2019)—which increases their likelihood to combine goals in a way that minimizes conflict and maximizes facilitation—and partly on situational constraints (e.g., what the other goals are and how they relate to each other). As it stands, there remain a lot of open questions in the literature on goal conflict resolution.…”