Assessing the state of software in a large enterprise

Abstract

To be relevant to the goals of an enterprise, an industrial software engineering research organization must identify problems of interest to, and find solutions that have an impact on, the software development organizations within the company. Using a systematic measurement program both to identify the problems and assess the impact of solutions is key to satisfying this need. Avaya has had such a program in place for about seven years. Every year we produce an annual report known as the State of Software in Avaya that describes software development trends throughout the company and that contains prioritized recommendations for improving Avaya's software development capabilities. We start by identifying the goals of the enterprise and use the goal-question-metric approach to identify the measures to compute. The result is insight into the enterprise's problems in software development, recommendations for improving the development process, and problems that require research to solve. We will illustrate the process with examples from the Software Technology Research Department in Avaya Labs whose purpose is to improve the state of software development and know it. "Know it" means that improvement should be subjectively evident and objectively quantifiable. "Know it" also means that one must be skilled at identifying the data sources, performing the appropriate analyses to answer the questions of interest, and validating that the data are accurate and appropriate for the purpose. Examples will include how and why we developed a measure of software quality that appeals to customers, how and why we are studying the effectiveness of distributed software development, and how and why we are helping development organizations to adopt iterative development methods. We will also discuss how we keep the company and the department apprised of the current strengths and weaknesses of software development in Avaya through the publication of the annual State Empir Software Eng (2010) 15: of Software in Avaya Report. Our purpose is both to provide a model for assessment that others may emulate, based on seven years of experience, and to spotlight analyses and conclusions that we feel are common to software development today.

Risky files: an approach to focus quality improvement effort

“…This is followed by a meeting discussing the initial analysis, and a training session for expert developers based on a focused analysis that is geared to the specific needs of the project and based on feedback from the initial meeting. The projects are identified by working with executives as part of communication about the state of software in Avaya [4]. Once a project is identified we conduct the relevant analysis.…”

Engineering big data solutions

“…ODS addressing strategic software development goals in an enterprise are presented in, for example, [20]. A survey of Microsoft managers and developers [4] provides a list of practical questions for ODS.…”

Evidence-based software production

“…A few attempts have been made to establish such standardization within individual organizations [7], [8]. A key research (and technology) issue is how to extend such attempts to cover entire industries and market segments.…”

“…We can detect trends, identify correlations, and then find causal links that we can use to improve our software production capabilities. As noted before, this has been done on smaller scales than we are proposing [7], [8], but has not been tried on much larger scales. Standardizing, collecting, maintaining, making available, and analyzing the data automatically and unobtrusively are not easy tasks, nor are they tasks that we can do right now.…”

“…It may be true that in some cases, particularly where there is little recording of knowledge now, that some initial development costs will be incurred. However, studies of large organizations, such as [7], indicate small overhead in measurement costs, partly because much of the quantitative data collection and analysis is automated.…”