Caret-HM: recording and replaying Android user sessions with heat map generation using UI state clustering

Nurmuradov, Dmitry; Bryce, Renée

doi:10.1145/3092703.3098231

Cited by 11 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nurmuradov et al [80] introduced a record and replay tool for Android applications that captures user interactions by displaying the device screen in a web browser. This technique uses event data captured during the recording process to generate a heatmap that facilitate developers' understanding on how users are interacting with an application.…”

Section: Record and Replaymentioning

confidence: 99%

Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios

Bernal-Cárdenas

Cooper

Havranek

et al. 2023

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Screen recordings of mobile applications are easy to obtain and capture a wealth of information pertinent to software developers (e.g., bugs or feature requests), making them a popular mechanism for crowdsourced app feedback. Thus, these videos are becoming a common artifact that developers must manage. In light of unique mobile development constraints, including swift release cycles and rapidly evolving platforms, automated techniques for analyzing all types of rich software artifacts provide benefit to mobile developers. Unfortunately, automatically analyzing screen recordings presents serious challenges, due to their graphical nature, compared to other types of (textual) artifacts. To address these challenges, this paper introduces V2S+, an automated approach for translating video recordings of Android app usages into replayable scenarios. V2S+ is based primarily on computer vision techniques and adapts recent solutions for object detection and image classification to detect and classify user gestures captured in a video, and convert these into a replayable test scenario. Given that V2S+ takes a computer vision-based approach, it is applicable to both hybrid and native Android applications. We performed an extensive evaluation of V2S+ involving 243 videos depicting 4,028 GUI-based actions collected from users exercising features and reproducing bugs from a collection of over 90 popular native and hybrid Android apps. Our results illustrate that V2S+ can accurately replay scenarios from screen recordings, and is capable of reproducing ≈ 90.2% of sequential actions recorded in native application scenarios on physical devices, and ≈ 83% of sequential actions recorded in hybrid application scenarios on emulators, both with low overhead. A case study with three industrial partners illustrates the potential usefulness of V2S+ from the viewpoint of developers.

show abstract

Section: Record and Replaymentioning

confidence: 99%

Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios

Bernal-Cárdenas

Cooper

Havranek

et al. 2023

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…Nurmuradov et al [57] introduced a record and replay tool for Android applications that captures user interactions by displaying the device screen in a web browser. It used event data captured during the recording process to generate a heatmap that facilitates developers understanding of how users are interacting with an application.…”

Section: Bug Record and Replay From App Running Informationmentioning

confidence: 99%

GIFdroid: Automated Replay of Visual Bug Reports for Android Apps

Feng¹,

Chen²

2021

Preprint

View full text Add to dashboard Cite

Bug reports are vital for software maintenance that allow users to inform developers of the problems encountered while using software. However, it is difficult for non-technical users to write clear descriptions about the bug occurrence. Therefore, more and more users begin to record the screen for reporting bugs as it is easy to be created and contains detailed procedures triggering the bug. But it is still tedious and time-consuming for developers to reproduce the bug due to the length and unclear actions within the recording. To overcome these issues, we propose GIFdroid, a lightweight approach to automatically replay the execution trace from visual bug reports. GIFdroid adopts image processing techniques to extract the keyframes from the recording, map them to states in GUI Transitions Graph, and generate the execution trace of those states to trigger the bug. Our automated experiments and user study demonstrate its accuracy, efficiency, and usefulness of the approach. CCS CONCEPTS• Software and its engineering → Software testing and debugging.

show abstract

“…Nurmuradov et al [61] introduced a record and replay tool for Android applications that captures user interactions by displaying the device screen in a web browser. This technique uses event data captured during the recording process to generate a heatmap that facilitate developers' understanding on how users are interacting with an application.…”

Section: Related Workmentioning

confidence: 99%

Translating video recordings of mobile app usages into replayable scenarios

Bernal-Cárdenas

Cooper

Moran

et al. 2020

Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

View full text Add to dashboard Cite

Screen recordings of mobile applications are easy to obtain and capture a wealth of information pertinent to software developers (e.g., bugs or feature requests), making them a popular mechanism for crowdsourced app feedback. Thus, these videos are becoming a common artifact that developers must manage. In light of unique mobile development constraints, including swift release cycles and rapidly evolving platforms, automated techniques for analyzing all types of rich software artifacts provide benet to mobile developers. Unfortunately, automatically analyzing screen recordings presents serious challenges, due to their graphical nature, compared to other types of (textual) artifacts. To address these challenges, this paper introduces V2S, a lightweight, automated approach for translating video recordings of Android app usages into replayable scenarios. V2S is based primarily on computer vision techniques and adapts recent solutions for object detection and image classication to detect and classify user actions captured in a video, and convert these into a replayable test scenario. We performed an extensive evaluation of V2S involving 175 videos depicting 3,534 GUI-based actions collected from users exercising features and reproducing bugs from over 80 popular Android apps. Our results illustrate that V2S can accurately replay scenarios from screen recordings, and is capable of reproducing ⇡ 89% of our collected videos with minimal overhead. A case study with three industrial partners illustrates the potential usefulness of V2S from the viewpoint of developers. CCS CONCEPTS • Software and its engineering → Software maintenance tools; Software verication and validation; Application specic development environments.

show abstract

Caret-HM: recording and replaying Android user sessions with heat map generation using UI state clustering

Cited by 11 publications

References 12 publications

Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios

Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios

GIFdroid: Automated Replay of Visual Bug Reports for Android Apps

Translating video recordings of mobile app usages into replayable scenarios

Contact Info

Product

Resources

About