Sorry, I don’t Understand: Improving Voice User Interface Testing

Abstract

Context. The game industry is increasingly growing in recent years. Every day, millions of people play video games, not only as a hobby, but also for professional competitions (e.g., e-sports or speed-running) or for making business by entertaining others (e.g., streamers). The latter daily produce a large amount of gameplay videos in which they also comment live what they experience. But no software and, thus, no video game is perfect: Streamers may encounter several problems (such as bugs, glitches, or performance issues) while they play. Also, it is unlikely that they explicitly report such issues to developers. The identified problems may negatively impact the user's gaming experience and, in turn, can harm the reputation of the game and of the producer. Objective. In this paper, we propose and empirically evaluate GELID, an approach for automatically extracting relevant information from gameplay videos by (i) identifying video segments in which streamers experienced anomalies; (ii) categorizing them based on their type (e.g., logic or presentation); clustering them based on (iii) the context in which appear (e.g., level or game area) and (iv) on the specific issue type (e.g., game crashes). Method. We manually defined a training set for step 2 of GELID (categorization) and a test set for validating in isolation the four components of GELID. In total, we manually segmented, labeled, and clustered 170 videos related to 3 video games, defining a dataset containing 604 segments. Results. While in steps 1 (segmentation) and 4 (specific issue clustering) GELID achieves satisfactory results, it shows limitations on step 3 (game context clustering) and, above all, step 2 (categorization).

VITAS : Guided Model-based VUI Testing of VPA Apps

“…SkillExplorer and SkillDetective [8] adopt the DFS-based exploration approach. VUI-UPSET [12,13] is a chat-bot style testing approach to generate correct paraphrases while detecting bugs. Vitas [10] uses the model-based testing to test VPA apps' problems related to quality, privacy and security.…”

“…To detect such problems, a thorough exploration of VPA apps' behavior is necessary. Existing methods mainly employed strategies of depth-first search based chatbot-style testing [6,8,9,12,13] or model-based testing (MBT) [10].…”