VITAS : Guided Model-based VUI Testing of VPA Apps

Abstract

The flourishing ecosystem centered around voice personal assistants (VPA), such as Amazon Alexa, has led to the booming of VPA apps. The largest app market Amazon skills store, for example, hosts over 200,000 apps. Despite their popularity, the open nature of app release and the easy accessibility of apps also raise significant concerns regarding security, privacy and quality. Consequently, various testing approaches have been proposed to systematically examine VPA app behaviors. To tackle the inherent lack of a visible user interface in the VPA app, two strategies are employed during testing, i.e., chatbot-style testing and model-based testing. The former often lacks effective guidance for expanding its search space, while the latter falls short in interpreting the semantics of conversations to construct precise and comprehensive behavior models for apps.In this work, we introduce Elevate, a model-enhanced large language model (LLM)-driven VUI testing framework. Elevate leverages LLMs' strong capability in natural language processing to compensate for semantic information loss during model-based VUI testing. It operates by prompting LLMs to extract states from VPA apps' outputs and generate context-related inputs. During the automatic interactions with the app, it incrementally constructs the behavior model, which facilitates the LLM in generating inputs that are highly likely to discover new states. Elevate bridges the LLM and the behavior model with innovative techniques such as encoding behavior model into prompts and selecting LLM-generated inputs based on the context relevance. Elevate is benchmarked on 4,000 real-world Alexa skills, against the state-of-the-art tester Vitas. It achieves 15% higher state space coverage compared to Vitas on all types of apps, and exhibits significant advancement in efficiency.

SkillScanner: Detecting Policy-Violating Voice Applications Through Static Analysis at the Development Phase

“…VerHealth [53] analyzed 813 health-related skills on the Amazon Alexa platform. Vitas [42] interacted with 41,581 skills and found that 51% of skills suffered from problems such as unexpected exit/start and privacy violation. SkillDetective [59] identified 6,079 policy-violating skills in the current skills stores by evaluating skills' conformity to more than 50 different policy requirements.…”

“…Distinction from existing works. SkillScanner distincts itself from existing policy violation detection works (including SkillExplorer [39], VerHealth [53], Vitas [42] and SkillDetective [59]) in three ways. First, different from recent skill testing tools that utilize dynamic analysis and can only detect violations exercised by run-time inputs, SkillScanner is the first-of-its-kind static analysis tool and it detects more possible violations/inconsistencies in the code.…”

“…Although these policies are checked during the skill certification process (which rejects a skill if it violates any of the pre-defined policies), prior work demonstrated the ease of policyviolating skills being certified [32,55]. Several recent works [39,41,42,53,59] developed tools to measure the policy compliance of skills on the Amazon Alexa platform through a dynamic analysis approach (i.e., by exploring the outcomes of skills). For example, SkillExplorer [39] found 1,141 skills that collect different types of private information but without disclosing the data collection in their privacy policies.…”

On the Quality of Privacy Policy Documents of Virtual Personal Assistant Applications

“…The privacy concerns of the VPA app ecosystem have also raised high attention from the research community. A line of research has been dedicated to detecting the runtime information gathering behaviors of VPA apps [35,47,79]. Some recent studies [81,82] propose to examine the compliance between VPA apps' data handling behaviors and the statements in their privacy policies.…”