Automating performance testing of interactive Java applications

Abstract

Interactive applications with graphical user interfaces are prevalent in today's environment: Everybody with access to any kind of computer constantly uses them. A significant body of prior work has devised approaches for automating the functional testing of such applications. However, no such work exists for automatically testing their performance. Performance testing imposes additional requirements upon GUI test automation tools: the tools have to be able to replay complex interactive sessions, and they have to avoid perturbing the application's performance. We study the feasibility of using five Java GUI capture & replay tools for GUI performance test automation. Besides confirming the severity of the previously known GUI element identification problem, we also identify a related problem, the temporal synchronization problem, which is of increasing importance for GUI applications that use timer-driven activity. We find that most of the tools we study have severe limitations when used for recording and replaying realistic sessions of real-world Java applications, and that all of them suffer from the temporal synchronization problem. However, we find that the most reliable tool, Pounder, causes only limited perturbation, and thus can be used to automate performance testing. Besides the significance of our findings to GUI performance testing, the results are also relevant to capture & replay-based functional GUI test automation approaches.

RERAN: Timing- and touch-sensitive record and replay for Android

“…First, the user inputs the geographical location-Los Angeles-using the touchpad (seconds 0-10); notice how the compass is not used in this phase. Next, the app loads the gas station maps in the vicinity of Los Angeles and uses the compass to orient the map; notice how the user waits at this point as indicated by the absence of user events (seconds [11][12][13][14][15][16][17][18][19][20][21][22][23][24][25]. In the third phase, the map has been loaded and the user navigates on the map using pinch and zoom, indicated by the high number of touchscreen events (seconds 26-45); the compass is still used to keep the correct map orientation.…”

“…Currently, we do not capture such sensors, as tapping into such sources would require a non-trivial extension to our system, a task we leave to future work. Dynamic or random: Similarly, we do not capture sources of nondeterminism, for example arising through dynamic layouts or popup windows [11], network connectivity changes [12], or internal nondeterminism such as the use of a random number generator. As can be seen in Table VI, the majority of these dynamic or random apps are games, e.g., the popular game Temple Run randomly generates levels each time a user plays, making replay with RERAN impossible.…”

“…These tools, even though they are at a higher level of abstraction than RERAN, are also susceptible to errors when changes in the GUI occur, e.g., Button1 changes position between application versions. Some of these tools also offer support for mouse drags [11]. However, mouse drags differ from swipes in that the individual events being sent by other platforms are usually being detected as mouse presses (which already contain positioning information).…”

Combining profiling and monitoring to analyze test coverage and identify performance problems

“…There is some existing work on combining profiling and monitoring [5]. Jovic et al describe an approach for the automation of performance testing of interactive java applications [4].…”

Open source capture and replay tools comparison

“…There are some previous work on this topic, such as [3], the one that we use as our starting point, however our contribution is different, because we also compare the tools in terms of user experience, involving ease of learn and ease of use, which is not done in the referenced paper.…”