Skoll: A Process and Infrastructure for Distributed Continuous Quality Assurance

Abstract

Abstract-Software engineers increasingly emphasize agility and flexibility in their designs and development approaches. They increasingly use distributed development teams, rely on component assembly and deployment rather than green field code writing, rapidly evolve the system through incremental development and frequent updating, and use flexible product designs supporting extensive end-user customization. While agility and flexibility have many benefits, they also create an enormous number of potential system configurations built from rapidly changing component implementations. Since today's quality assurance (QA) techniques do not scale to handle highly configurable systems, we are developing and validating novel software QA processes and tools that leverage the extensive computing resources of user and developer communities in a distributed, continuous manner to improve software quality significantly. This paper provides several contributions to the study of distributed, continuous QA (DCQA). First, it shows the structure and functionality of Skoll, which is an environment that defines a generic around-the-world, around-the-clock QA process and several sophisticated tools that support this process. Second, it describes several novel QA processes built using the Skoll environment. Third, it presents two studies using Skoll: one involving user testing of the Mozilla browser and another involving continuous build, integration, and testing of the ACE+TAO communication software package. The results of our studies suggest that the Skoll environment can manage and control distributed continuous QA processes more effectively than conventional QA processes. For example, our DCQA processes rapidly identified problems that had taken the ACE+TAO developers much longer to find and several of which they had not found. Moreover, the automatic analysis of QA results provided developers information that enabled them to quickly find the root causes of problems.

Using symbolic evaluation to understand behavior in configurable software systems

“…We then performed substantial analysis on the results to explore the configuration space of each subject program. To do this we used the Skoll system, developed and housed at UMD [12]. Skoll allows users to define configurable QA tasks and run them across large virtual computing grids.…”

Testing Adaptive Probabilistic Software Components in Cyber Systems

“…The usefulness of sound and systematic approaches for achieving dependable, flexible, and cost-effective software has been a focus of scientific interest for many years [16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35]. As we move to automatically adaptive software, these types of approaches are appearing to become a necessity rather than a luxury, because manual artisanship simply cannot reach the levels of reliability and speed of adaptation that is called for.…”

“…The cost for this step is proportional to the number of components, and the process must be done once for each version of each atomic component. Technologies for doing this are known, and some of them are starting to be used in practice [30]. 3.…”

Test Algebra for Concurrent Combinatorial Testing

“…CAs reveal faults that arise from improper interaction of t or fewer elements Porter et al (2007). The strength of CA is important for testing.…”