Model Based Testing in Practice at Microsoft

Abstract

As part of Microsofts Trustworthy Computing [4] initiative the company has sought many ways to increase reliability. One approach being extensively investigated and used is Model Based Testing. With a Finite State Machine modeling tool (TMT) successfully deployed and in use by many test groups, a need for more powerful and flexible modeling has arisen. Several product groups are exploring the use of the Abstract State Machine Language (AsmL) and its associated test tool (AsmL/T). Results from both approaches have shown an increased ability to find defects earlier, including in the specification and design stages, as well as achieve higher structural code coverage on the actual systems under test.

Reducing the Costs of Bounded-Exhaustive Testing

“…The experience with bounded-exhaustive testing in academia and industry shows that it can find faults in real code [4,3,5,11] but also produces a large number of failures. Identifying a few faults out of many failures is a challenging task.…”

“…Previous work considered how to describe a set of inputs (using declarative [2,4] or imperative [5] approaches) and how to efficiently generate them. Bounded-exhaustive testing has been successfully used to reveal faults in several real applications [3,4,5,11], but it has costs in machine time for test generation and execution and human time for inspection of failures. This paper presents three new techniques that reduce the costs of such testing.…”

“…Our focus is on boundedexhaustive testing [2,3,4,5,6,11] that tests the code for all inputs within given bounds. Previous work considered how to describe a set of inputs (using declarative [2,4] or imperative [5] approaches) and how to efficiently generate them.…”

“…Two key costs in this context are machine time for test generation and execution (which also translates into human time for waiting for these results [9,10]) and human time for inspection of failures. Previous experience shows that bounded-exhaustive testing can discover important faults [3,4,5,11] but also can have high costs. This paper proposes, and evaluates on a case study, three novel techniques that reduce these costs of bounded-exhaustive testing.…”

Optimizing Generation of Object Graphs in Java PathFinder

“…We developed an approach [6,[24][25][26] for automated, constraint-based test generation: the user manually encodes only the constraints for the desired properties of test inputs, and a tool automatically generates a large number of inputs. Our approach was used to test several applications, e.g., a network protocol and a constraint analyzer developed in academia [25], a fault-tree analyzer developed for NASA [30], an XPath compiler at Microsoft [29], and a web traversal application at Google [26]. Generation tools for our approach were mostly developed from scratch.…”