Performance assertion checking

Abstract

Abstract. Performance assertion checking is an approach to describing and monitoring the performance of complex software systems. The idea is simple: system implementors write assertions that capture their expectations for performance, the system is instrumented to collect performance data, and then the assertions are checked automatically against the data to detect violations signifying potential performance bugs. Because performance assertions provide a means of ltering data based on expectations, they form a good basis for tools. Data indicating that a system is performing as expected can be discarded automatically, while data indicating potential problems can be brought to the attention of a person.Performance assertions are useful for performance regression testing, continuous system monitoring, and performance debugging. Also, the act of writing precise performance assertions helps designers and implementors to understand better their own expectations for performance, and the guarantees they can make about their systems.PSpec is a language and a set of tools that embodies the idea of performance assertion checking. A PSpec user writes performance assertions in the PSpec language. These are input, along with a monitoring log, to a checker tool that reports which assertions fail to hold for the log. Another tool called the solver helps to ll in constants in assertions using data in a monitoring log. PSpec has been used to nd performance bugs in the runtime system of a new parallel programming language, providing evidence of its utility.

Asserting Performance Expectations

“…Many research efforts have modeled the performance properties of applications [6,11]. In fact, the name of performance assertions is not in and of itself novel.…”

“…However, our technique and prototype, which are novel, allow users to assert explicitly in their code their performance properties, which can be verified empirically at runtime. In contrast to earlier work by Perl [11], this research focuses on runtime techniques to judge if an assertion has met its expectation. Perl's work checked for these properties in event logs, not in the application at runtime.…”

Performance assertions for mobile devices

“…Because assertions became a wellknown and easy to use tool, researchers and practitioners tried to extend them for validation of non-functional requirements, such as performance. Perl [9] [10] proposed to use assertions for validation of performance requirements and implemented such performance assertion system. Perl's work concentrated on post-mortem assertion checking using trace logs.…”

An integrated environment for the complete development cycle of an object-oriented distributed real-time system

“…In many published monitoring systems [8,10,11], the required instrumentation has to be inserted manually. In addition, often events have to be named explicitly and this naming has to be done manually as well.…”