Exploiting the Saturation Effect in Automatic Random Testing of Android Applications

Abstract

Monkey Fuzz Testing (MFT), a form of random testing, continues to gain popularity to test Android apps because of its ease of use. (Untrained) programmers use MFT tools to fully automatically detect certain classes of faults in apps. A challenge for these tools is the lack of a stopping criterion---programmers currently typically stop these tools when they run out of time. In this paper, we use the notion of the Saturation Effect of an MFT tool on an app under test to define a stopping criterion, parameterized by the app's preconditions and the tool's configurations. We have implemented our approach in the AndroidRipper MFT tool. We experimentally report results on 18 real Android app subjects. We show that the saturation effect is able to stop testing when test adequacy has been achieved without wasting test cycles

How do Developers Test Android Applications?

“…The ideal is to have a tool able to perform an entire session (run several scenarios) / without having the test-runner to kill and initialize the app before running each scenario." Although random testing has been proven to be "useful" by the research community [14], [19], [24], [26], available random testing tools for mobile apps (e.g., Monkey) have issues such as lack of expressiveness. For instance, the Android Monkey tool allows for reproducibility of event streams (by using the same seed), but it does not have log capabilities for creating a higher level representation of the streams.…”

Random GUI Testing of Android Application Using Behavioral Model

“…We also plan on extending our tool based on the limitations given in section V. We intend to evaluate our tool with larger Android application in the Google Play. Finally, we will further investigate our tool in term of time to reach the saturation point [18], and ability to find bugs.…”

ChimpCheck: property-based randomized test generation for interactive apps

“…Testing this app requires using test fixtures for getting past user authentication and asserting properties of the user interface specific to being a music player. For concreteness, let us consider testing a particular user story for the app: (1) Click on button Enter; (2) Type in test and 1234 into the username and password text boxes, respectively; (3) Click on button Sign in; and (4) Click on buttons ○ and to try out starting and stopping the music Figure 2. ChimpCheck focuses test development to specifying the skeleton of user interactions-here, the valid and invalid sign-in flows for the music service app from Figure 1.…”

“…Much less attention was given to the programmability or customizability question to, for example, empower the test developer to express her own customized genes. In the end, studies [2,4,12] on the saturation and coverage of automatic test-case generation for Android apps provide evidence for the ChimpClick claim-the need for human insight and appspecific knowledge to generate not just traces but relevant traces.…”

“…Particularly, the key challenge for (1) is to formally define the semantics of interaction between the scripts that the developer writes and each specific test generation technique in which we fuse. For (2), the key challenges are developing general means of expressing multiple input domains (not only UI traces) and defining the means in which these input streams interact with one another. Generalizing reified input domains also introduces an opportunity for exploring an augmentation on state-of-the-art test generation techniques: generalizing them for generating not just UI traces but also other input sequences (e.g., GPS location updates, event-based broadcasts from other apps).…”