uxSense: Supporting User Experience Analysis with Visualization and Computer Vision

Batch, Andrea; Ji, Yipeng; Fan, Mingming; Zhao, Jian; Elmqvist, Niklas

doi:10.1109/tvcg.2023.3241581

Cited by 7 publications

(4 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Additionally, expert feedback suggested that the SI model for narrative maps held potential value for analysts in their sense-making processes. Batch et al [19] introduced uxSENSE, a visual analytics system leveraging machine learning techniques to extract user behavior from audio and video recordings. The proposed method extracts multi-modal features of human behavior, including user sentiment, actions, posture, spoken words, and other relevant features, from such recordings.…”

Section: Capturing and Analyzing Semantic User Interactions In Visual...mentioning

confidence: 99%

Leveraging Machine Learning to Analyze Semantic User Interactions in Visual Analytics

Jeong,

2024

Information

View full text Add to dashboard Cite

In the field of visualization, understanding users’ analytical reasoning is important for evaluating the effectiveness of visualization applications. Several studies have been conducted to capture and analyze user interactions to comprehend this reasoning process. However, few have successfully linked these interactions to users’ reasoning processes. This paper introduces an approach that addresses the limitation by correlating semantic user interactions with analysis decisions using an interactive wire transaction analysis system and a visual state transition matrix, both designed as visual analytics applications. The system enables interactive analysis for evaluating financial fraud in wire transactions. It also allows mapping captured user interactions and analytical decisions back onto the visualization to reveal their decision differences. The visual state transition matrix further aids in understanding users’ analytical flows, revealing their decision-making processes. Classification machine learning algorithms are applied to evaluate the effectiveness of our approach in understanding users’ analytical reasoning process by connecting the captured semantic user interactions to their decisions (i.e., suspicious, not suspicious, and inconclusive) on wire transactions. With the algorithms, an average of 72% accuracy is determined to classify the semantic user interactions. For classifying individual decisions, the average accuracy is 70%. Notably, the accuracy for classifying ‘inconclusive’ decisions is 83%. Overall, the proposed approach improves the understanding of users’ analytical decisions and provides a robust method for evaluating user interactions in visualization tools.

show abstract

Section: Capturing and Analyzing Semantic User Interactions In Visual...mentioning

confidence: 99%

Leveraging Machine Learning to Analyze Semantic User Interactions in Visual Analytics

Jeong,

2024

Information

View full text Add to dashboard Cite

show abstract

“…Additionally, expert feedback suggested that the SI model for narrative maps holds potential value for analysts in their sensemaking processes. Batch et al [18] introduced uxSENSE, a visual analytics system leveraging machine learning techniques to extract user behavior from audio and video recordings. The proposed method extracts multi-modal features of human behavior, including user sentiment, actions, posture, spoken words, and other relevant features, from such recordings.…”

Section: Capturing and Analyzing Semantic User Interactions In Visual...mentioning

confidence: 99%

Understanding Semantic User Interactions in Visualization

Jeong,

2024

Preprint

View full text Add to dashboard Cite

In the field of visualization, understanding users’ analytical process is important to determine the effectiveness of a designed visualization application. To understand the users’ analytical process, numerous studies have been performed to capture and analyze user interactions. Although many studies have emphasized the importance of analyzing user interactions to understand users’ analytical reasoning processes, few have successfully linked these interactions to users’ reasoning processes. This paper introduces an approach that bridges this gap by correlating semantic user interactions with analysis decisions through an interactive wire transaction analysis system and a visual state transition matrix. With the designed analysis system, interactive analysis can be performed to evaluate financial fraud in wire transactions. It also allows the mapping of captured user interactions and analytical decisions back onto the visualization, revealing users’ distinct results. The visual state transition matrix further aids to help understanding users’ analytical flows, revealing their decision-making processes. Classification machine learning algorithms are applied to assess the effectiveness of our approach in understanding analysts’ strategies by connecting them to their decisions. From the study, we observed an average of 72% accuracy in clearly classifying the semantic user interactions. For classifying individual decisions, we observed an average of 70% classification accuracy. This emphasizes the importance of capturing semantic user interactions to understand users’ analytical processes. Overall, it is determined that the proposed approach not only enhances the understanding of analytical behaviors but also offers a robust approach for evaluating user interactions in visualization tools.

show abstract

“…Computer vision techniques involve analyzing and understanding digital images and videos with the help of algorithms and mathematical models. (Batch et al, 2023) propose UXSense -a method for extracting multi-modal features of human behavior from video and audio footage using machine learning (ML) to support UX and usability professionals in their analysis of user session data using interactive visualization. Furthermore, computer vision in video analysis has gained tremendous attention in recent years.…”

Section: Ux and Machine Learning -Preprocessing Of Audiovisual Data U...mentioning

confidence: 99%

UX and Machine Learning – Preprocessing of Audiovisual Data Using Computer Vision to Recognize UI Elements

Čejka,

Masner,

Jarolímek

et al. 2023

AOL

View full text Add to dashboard Cite

This study explores the convergence of user experience (UX) and machine learning, particularly employing computer vision techniques to preprocess audiovisual data to detect user interface (UI) elements. With an emphasis on usability testing, the study introduces a novel approach for recognizing changes in UI screens within video recordings. The methodology involves a sequence of steps, including form prototype creation, laboratory experiments, data analysis, and computer vision tasks. The future aim is to automate the evaluation of user behavior during UX testing. This innovative approach is relevant to the agricultural domain, where specialized applications for precision agriculture, subsidy requests, and production reporting demand streamlined usability. The research introduces a frame extraction algorithm that identifies screen changes by analyzing pixel differences between consecutive frames. Additionally, the study employs YOLOv7, an efficient object detection model, to identify UI elements within the video frames. Results showcase successful screen change detection with minimal false negatives and acceptable false positives, showcasing the potential for enhanced automation in UX testing. The study’s implications lie in simplifying analysis processes, enhancing insights for design decisions, and fostering user-centric advancements in diverse sectors, including precision agriculture.

show abstract

uxSense: Supporting User Experience Analysis with Visualization and Computer Vision

Cited by 7 publications

References 61 publications

Leveraging Machine Learning to Analyze Semantic User Interactions in Visual Analytics

Leveraging Machine Learning to Analyze Semantic User Interactions in Visual Analytics

Understanding Semantic User Interactions in Visualization

UX and Machine Learning – Preprocessing of Audiovisual Data Using Computer Vision to Recognize UI Elements

Contact Info

Product

Resources

About