Main effects screening: a distributed continuous quality assurance process for monitoring performance degradation in evolving software systems

Yılmaz, Cemal; Krishna, Arvind S.; Memon, Atif M.; Porter, Adam; Schmidt, Douglas C.; Gokhale, A.; Natarajan, Balachandran

doi:10.1109/icse.2005.1553572

Cited by 22 publications

(24 citation statements)

References 11 publications

(3 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We evaluated the main effects screening process via several industrial strength feasibility studies on ACE+TAO. Our results indicate that main effects screening can reliably and accurately detect key sources of performance degradation in large-scale systems with significantly less effort than conventional techniques [Skoll05].…”

Section: Ensuring Coherency and Reducing Redundancy In Qa Activitiesmentioning

confidence: 87%

“…For example, our configuration model essentially defines a combinatorial object against which a wide variety of statistical tools can be applied. In another case study [Skoll05], we leveraged this feature to develop a DCQA process called main effects screening for monitoring performance degradations in evolving systems.…”

Section: Ensuring Coherency and Reducing Redundancy In Qa Activitiesmentioning

confidence: 99%

See 1 more Smart Citation

Techniques and processes for improving the quality and performance of open‐source software

Porter

Yılmaz

Memon

et al. 2006

Softw Process Imprv Pract

View full text Add to dashboard Cite

Section: Ensuring Coherency and Reducing Redundancy In Qa Activitiesmentioning

confidence: 87%

Section: Ensuring Coherency and Reducing Redundancy In Qa Activitiesmentioning

confidence: 99%

Techniques and processes for improving the quality and performance of open‐source software

Porter

Yılmaz

Memon

et al. 2006

Softw Process Imprv Pract

View full text Add to dashboard Cite

“…The results indicated that (1) this process cheaply and correctly identifies the subset of options that are most important to system performance, (2) monitoring only these selected options can detect performance degradation quickly with an acceptable level of effort, and (3) alternative strategies with equivalent effort yield less reliable results. See [32,33] for more details. An interesting aspect of this approach is that by computing the key effects before changes occur, we cut down total benchmarking time from 2 days to 5 minutes, which is fast enough to make this part of the source code check-in process.…”

Section: Summary Of Prior Workmentioning

confidence: 99%

“…In prior work [13,24,34], we developed a prototype DCQA environment called Skoll [27] that improves upon earlier system approaches described in Section 1.1. In particular, Skoll provides an Intelligent Steering Agent (ISA) that guides the QA process across large configuration spaces by decomposing QA analyses (such as anomaly detection, QoS evaluation, and integration testing QA processes) into multiple tasks and then distributing/executing these tasks continuously across a grid of computing resources contributed by end-users and distributed developers around the world.…”

Section: Summary Of Prior Workmentioning

confidence: 99%

Towards a Distributed Continuous Certification Process

Porter

2007

2007 IEEE International Parallel and Distributed Processing Symposium

View full text Add to dashboard Cite

Software scale and complexity are growing by every measure: more hardware and software, more communication links, more interdependency, more lines of code, more storage and data, etc. [18]. At the same time business trends are increasingly squeezing development resources. In particular, development processes are straining under severe cost and time-to-market pressures. Global competition and market deregulation are shrinking profit margins and thus limiting budgets for the development and QA of software.In response to these trends, developers have begun to change the way they build and validate software systems by (among other things) moving towards more flexible product designs allowing dynamic reconfiguration.This approach promises to improve cost, quality, and development-time, but creates other problems, especially when used in the context of safety-critical systems.To realize this promise, however, effective certification becomes more important than ever since as static controls are removed or reduced, it becomes even more vital that (1) problems be caught as quickly as possible and (2) systems not be allowed to drift so far from their intended functional and performance requirements that rework costs overwhelm the hoped-for efficiencies.This article will present and discuss some of our recent efforts to address these problems.

show abstract

“…It could contain other sub-components, but such architectural information would not be used in this focus area. In our own research we have started by developing a system and approach currently called QUASI (Quality as a Service Infrastructure) [11,5,6,9]. QUASI's analytical cornerstone is a model of the design space that implicitly captures all configurations on which test jobs might run.…”

Section: Testing Individual Componentsmentioning

confidence: 99%

Community-based, collaborative testing and analysis

Memon

Porter

Sussman

2010

Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research

View full text Add to dashboard Cite

This article proposes a research agenda aimed at enabling optimized testing and analysis processes and tools to support component-based software development communities. We hypothesize that de facto communities-sets of projects that provide, maintain and integrate many shared infrastructure components-are commonplace. Currently, community members, often unknown to each other, tend to work in isolation, duplicating work, failing to learn from each other's effort, and missing opportunities to efficiently improve the common infrastructure. We further hypothesize that as software integration continues to become the predominant mode of software development, there will be increasing value in tools and techniques that empower these communities to coordinate and optimize their development efforts, and to generate and broadly share information. Such tools and techniques will greatly improve the robustness, quality and usability of the common infrastructure which, in turn, will greatly reduce the time and effort needed to produce and use the end systems that are the true goal of the entire community.

show abstract

Main effects screening: a distributed continuous quality assurance process for monitoring performance degradation in evolving software systems

Cited by 22 publications

References 11 publications

Techniques and processes for improving the quality and performance of open‐source software

Techniques and processes for improving the quality and performance of open‐source software

Towards a Distributed Continuous Certification Process

Community-based, collaborative testing and analysis

Contact Info

Product

Resources

About