Continuous Experimentation in Mobile Game Development

Abstract

Software companies need capabilities to evaluate the user value and the success of their products. This is especially crucial for highly competitive markets, such as the mobile game industry, where thousands of new games are introduced every month. Game companies often run continuous experiments as an integrated part of the overall development process. This paper presents a game company's journey on experimentation, and describes how the experiments are used at different stages of the development cycle to produce reliable, meaningful data for developers as well as how to balance between different data collection methods. Our study indicates that experiments are important in all stages of the development in different forms. Early stages in the development experiments can be run with proxy users due to lack of real users, whereas later in the development Key Performance Indicator (KPI) metrics play the most important role in experiments. Establishing concrete goals for the experiments, balancing between qualitative and quantitative data collection, experimentation throughout the development process with the guidance of an efficient leadership appears to be the key to success.

Controlled Experimentation in Continuous Experimentation: Knowledge and Challenges

“…Thus, there is a need for startup companies to be more involved with experimentation, although they have less infrastructure in place. Finally, we would like to call attention to some of the few case studies and experience reports on experimentation on "ordinary" software companies, which are neither multi-national corporations nor startups [46,63,64]; in e-commerce, customer relations, and gaming industry respectively. None of these papers are focused on infrastructure, but do mention that infrastructure needs to be implemented.…”

“…The app marketplaces impose constraints on deployment and variability. [131,96,64] Cyber-physical systems…”

“…Xu et al [96] at LinkedIn instead advocate the use of quasi-experimental designs. Finally, Yaman et al [64] have done an interview study on continuous experimentation, where they emphasize user feedback in the earlier stages of development (that do not require deployment).…”