To the attention of mobile software developers: guess what, test your app!

Abstract

Software testing is an important phase in the software development lifecycle because it helps in identifying bugs in a software system before it is shipped into the hand of its end users. There are numerous studies on how developers test general-purpose software applications. The idiosyncrasies of mobile software applications, however, set mobile apps apart from general-purpose systems (e.g., desktop, stand-alone applications, web services). This paper investigates working habits and challenges of mobile software developers with respect to testing. A key finding of our exhaustive study, using 1000 Android apps, demonstrates that mobile apps are still tested in a very ad hoc way, if tested at all. However, we show that, as in other types of software, testing increases the quality of apps (demonstrated in user ratings and number of code issues). Furthermore, we find evidence that tests are essential when it comes to engaging the community to contribute to mobile open source software. We discuss reasons and potential directions to address our findings. Yet another relevant finding of our study is that Continuous Integration and Continuous Deployment (CI/CD) pipelines are rare in the mobile apps world (only 26% of the apps are developed in projects employing CI/CD) -we argue that one of the main reasons is due to the lack of exhaustive and automatic testing.

Test automation in open-source Android apps

“…Empirical studies on mobile app testing. Previously, researchers have investigated how test automation is practically adopted [12,13,33,37,42,43,50]. Kochhar et al [37] analyzed over 600 Android apps on F-Droid to check the presence of test cases and computed the code coverage.…”

“…They also conducted surveys to understand the usage of automated testing tools and the challenges faced by developers while testing. Cruz et al [13] analyzed 1,000 Android apps on F-Droid to check their usage of automated testing frameworks and continuous integration tools. They also found that projects using automated testing have more contributors and commits on GitHub.…”

“…This annotation is used by JUnit-based testing frameworks, including both unit and UI testing frameworks such as JUnit [34], Robolectric [52], Mockito [46], and Espresso [26]. A prior study investigating the usage of testing frameworks in 1, 000 apps on F-Droid [13] shows that 100% of the adopted unit testing frameworks and 97% of the UI testing frameworks are JUnit-based. Furthermore, we classify a test case as a An assumption of our study is that the subject apps were developed with Android Studio.…”

“…To understand the test automation culture prevalent among mobile app developers, researchers have investigated the extent to which test automation is adopted in practice [12,13,33,37,42]. However, those studies are limited in terms of both scale and quality of the curated dataset.…”

Automated tests for cross‐platform mobile apps in multiple configurations

“…There are several UI testing frameworks for mobile apps; notable examples are Espresso and Robotium for native Android apps, XCTest for iOS, and Appium for cross‐platform apps. With such frameworks, practitioners code and maintain scripts that automate the execution of UI tests; this scenario seems to be the current state‐of‐the‐practice in automated UI testing of mobile apps [31, 33]. We surmise that our approach would require a similar effort as coding native UI tests.…”