It Takes Two to Tango: Combining Visual and Textual Information for Detecting Duplicate Video-Based Bug Reports

Abstract

When a bug manifests in a user-facing application, it is likely to be exposed through the graphical user interface (GUI). Given the importance of visual information to the process of identifying and understanding such bugs, users are increasingly making use of screenshots and screen-recordings as a means to report issues to developers. However, when such information is reported en masse, such as during crowd-sourced testing, managing these artifacts can be a time-consuming process. As the reporting of screen-recordings in particular becomes more popular, developers are likely to face challenges related to manually identifying videos that depict duplicate bugs. Due to their graphical nature, screen-recordings present challenges for automated analysis that preclude the use of current duplicate bug report detection techniques. To overcome these challenges and aid developers in this task, this paper presents TA N G O , a duplicate detection technique that operates purely on video-based bug reports by leveraging both visual and textual information. TANGO combines tailored computer vision techniques, optical character recognition, and text retrieval. We evaluated multiple configurations of Ta n g o in a comprehensive empirical evaluation on 4,860 duplicate detection tasks that involved a total of 180 screenrecordings from six Android apps. Additionally, we conducted a user study investigating the effort required for developers to manually detect duplicate video-based bug reports and compared this to the effort required to use TA N G O . The results reveal that TA N G O 's optimal configuration is highly effective at detecting duplicate video-based bug reports, accurately ranking target duplicate videos in the top-2 returned results in 83% of the tasks. Additionally, our user study shows that, on average, TANGO can reduce developer effort by over 60%, illustrating its practicality.

It Takes Two to Tango: Combining Visual and Textual Information for Detecting Duplicate Video-Based Bug Reports

“…1 calculates the maximum possible overlap that can occur if the videos were exact matches, summing across the similarity score of each frame pair. Our online appendix contains the detailed f/w-LCS algorithms with examples [42].…”

“…A task is successful if at least one duplicate video is found in the top-k results returned by TANGO. We report HIT@k for cut points k = 1-2 in this paper, and 1-10 in our online appendix [42].…”

“…In this section, we present the results for TANGO's best performing configurations. However, complete results can be found in our online appendix [42]. Table I shows the results for TANGO vis and TANGO txt when using SimCLR, SIFT, as the visual feature extractor, and OCR as the textual extractor.…”

“…1) TANGO, a duplicate detection approach for video-based bug reports of mobile apps which is able to accurately suggest duplicate reports; 2) The results of a comprehensive empirical evaluation that measures the effectiveness of TANGO in terms of suggesting candidate duplicate reports; 3) The results of a user study with experienced programmers that illustrates TANGO's practical applicability by measuring its potential for saving developer effort; and 4) A benchmark (included in our online appendix [42]) that enables (i) future research on video-based duplicate detection, bug replication, and mobile app testing, and (ii) the replicability of this work. The benchmark contains 180 video-based bug reports with duplicates, source code, trained models, duplicate detection tasks, TANGO's output, and detailed evaluation results.…”

AndroR2: A Dataset of Manually-Reproduced Bug Reports for Android apps

“…Another way to extend ANDROR2 is by including multiple different variants of bug reproduction scripts, i.e., those that perform different sets of actions that lead to the same bug, as opposed to the minimal action sequences currently provided. Such an extension could facilitate research which examines test case selection and prioritization techniques [34] as well as research on detecting duplicate video-based bug reports [35].…”