SeqDynamics: Visual Analytics for Evaluating Online Problem‐solving Dynamics

Abstract

Problem‐solving dynamics refers to the process of solving a series of problems over time, from which a student's cognitive skills and non‐cognitive traits and behaviors can be inferred. For example, we can derive a student's learning curve (an indicator of cognitive skill) from the changes in the difficulty level of problems solved, or derive a student's self‐regulation patterns (an example of non‐cognitive traits and behaviors) based on the problem‐solving frequency over time. Few studies provide an integrated overview of both aspects by unfolding the problem‐solving process. In this paper, we present a visual analytics system named SeqDynamics that evaluates students ‘problem‐solving dynamics from both cognitive and non‐cognitive perspectives. The system visualizes the chronological sequence of learners’ problem‐solving behavior through a set of novel visual designs and coordinated contextual views, enabling users to compare and evaluate problem‐solving dynamics on multiple scales. We present three scenarios to demonstrate the usefulness of SeqDynamics on a real‐world dataset which consists of thousands of problem‐solving traces. We also conduct five expert interviews to show that SeqDynamics enhances domain experts’ understanding of learning behavior sequences and assists them in completing evaluation tasks efficiently.

A Survey of Human‐Centered Evaluations in Human‐Centered Machine Learning

“…Papers that describe applications that do not allow user interaction with the model beyond filtering of data points (i.e., purely exploratory systems in which the user can neither influence the model behavior during the analysis session nor optimize towards a specific model output) (e.g., [JVW20; LLT∗20; XXL∗20]).…”

“… Papers that provide use cases or usage scenarios developed by the authors without the inclusion of expert feedback, or case studies that do not consider human factors pertaining to the expert (e.g., [GWGvW19; KTC∗19; LJLH19; PLM∗17; SJS∗18; WPB∗20]). This was the most frequent reason for exclusion. Papers that describe applications that do not allow user interaction with the model beyond filtering of data points (i.e., purely exploratory systems in which the user can neither influence the model behavior during the analysis session nor optimize towards a specific model output) (e.g., [JVW20; LLT∗20; XXL∗20]). Papers not describing system evaluations but research agendas (e.g., [AVW∗18]), or workshops (e.g., [AW18; BCP∗19]). Papers that provide quantitative evaluations of results not generated by participants in a study setting (e.g., [BZL∗18; YDP19]). …”

“…Qualitative research appears to be the favored approach, with 65% of the studies focusing on gathering qualitative feedback in the form of interviews, surveys, and observations, often leveraging think‐aloud protocols (e.g. [BSP20; CD19; HHC∗19; SDMT16; XXL∗20]. No significant change in trend across the years has been detected with quantitative data collection representing only 9% (e.g.…”