Transitioning Towards Continuous Experimentation in a Large Software Product and Service Development Organisation – A Case Study

Introducing continuous experimentation in large software-intensive product and service organisations

“…Relating to RQ3, the process of introducing continuous experimentations revealed several benefits for the companies as described in Section 5.3. Some of the benefits gained, which were also observed in related work Bosch, 2014 andYaman et al, 2016), include improved understanding of the need for user involvement in the development process and getting rapid feedback. Moreover, the teams gained new insights and understanding regarding their product and services, the users, and their development processes during the introduction.…”

“…Similarly to Company A, Company B has multiple layers of stakeholders involved with the product, as shown in Figure 2. We have previously described the details of the experiment conducted with this company in Yaman et al (2016).…”

“…In addition, because both companies were in the middle of developing an evolving service, there were existing feature prioritizations and deadlines that influenced the selection of the experiment target and scale of the experiments, as well as resource allocation. Together with the multilayered structure, this made it rather difficult to perform experiments that would have rather determined whether the features are at all necessary or suitable for accomplishing given tasks that have already been found to fulfill user needs (Yaman et al 2016). For Company B, this resulted in the selection of a small-scale UI experiment, even though a more ambitious or bigger experiment would have provided more value to the teams and the user.…”

Continuous Experimentation with Product-Led Business Models: A Comparative Case Study

“…Cycles of such experiments can lead to constant improvements in product quality and usability [5]. Continuous experimentation has been studied to see whether it is possible to apply in various challenging different software segments [15], such as for automotives where there are technical obstacles to overcome [8], or businessto-business companies where there are organizational obstacles [14] and possibly a lack of access to user data [23].…”

Notifying and Involving Users in Experimentation: Ethical Perceptions of Software Practitioners

“…Complex and uncertain customer requirements, and rapidly advancing technology, create pressure to deliver and evolve products fast. Hence, many software companies have adopted or are transitioning towards continuous experimentation, making product development decisions based on direct observation of user behaviour in their products [16], [4], [20], [23], [5].…”