Triage

Abstract

Diagnosing production run failures is a challenging yet important task. Most previous work focuses on offsite diagnosis, i.e. development site diagnosis with the programmers present. This is insufficient for production-run failures as: (1) it is difficult to reproduce failures offsite for diagnosis; (2) offsite diagnosis cannot provide timely guidance for recovery or security purposes; (3) it is infeasible to provide a programmer to diagnose every production run failure; and (4) privacy concerns limit the release of information (e.g. coredumps) to programmers.To address production-run failures, we propose a system, called Triage, that automatically performs onsite software failure diagnosis at the very moment of failure. It provides a detailed diagnosis report, including the failure nature, triggering conditions, related code and variables, the fault propagation chain, and potential fixes. Triage achieves this by leveraging lightweight reexecution support to efficiently capture the failure environment and repeatedly replay the moment of failure, and dynamically-using different diagnosis techniques-analyze an occurring failure. Triage employs a failure diagnosis protocol that mimics the steps a human takes in debugging. This extensible protocol provides a framework to enable the use of various existing and new diagnosis techniques. We also propose a new failure diagnosis technique, delta analysis, to identify failure related conditions, code, and variables.We evaluate these ideas in real system experiments with 10 real software failures from 9 open source applications including four servers. Triage accurately diagnoses the evaluated failures, providing likely root causes and even the fault propagation chain, while keeping normal-run overhead to under 5%. Finally, our user study of the diagnosis and repair of real bugs shows that Triage saves time (99.99% confidence), reducing the total time to fix by almost half.

Concurrency Debugging with Differential Schedule Projections

“…Triage [Tucek et al 2007] uses dynamic slicing to diagnose failures at the user's site, which obviates privacy concerns. Despite that, it has limited support for concurrency bugs, being able to provide root cause isolation only for multithreaded programs running on uniprocessors.…”

An empirical analysis of error propagation in critical software systems

“…An open-source flight simulator has been used to assess the proposal. In Tucek et al (2007) authors propose a system, called Triage, that automatically performs onsite software failure diagnosis. The system makes use of both kernel-level components and multiple re-executions of the target software to support failure diagnosis; during each re-execution, detailed data are collected via dynamic binary instrumentation to conduct the analysis of occurred failure and its causes.…”

“…In addition, the approach in Hiller et al (2004) requires measuring the error permeability for each input of each module, leading to a low scalability of the approach; while the tool (Hiller et al 2002a) addresses only single process software. The system proposed in Tucek et al (2007) uses kernel-level components and dynamic binary instrumentation, which is not allowed in critical production environments (e.g., mission critical systems) with stringent constraints imposed by certification standards and the use of obsolete kernel versions. Finally, the approaches (Hiller et al 2004;2002a;Johansson and Suri 2005) only address data errors, while those presented in Johansson and Suri (2005) and Calhoun et al (2017) are conceived only for OS device drivers and MPI applications, respectively.…”

“…To this aim, many existing approaches rely on quite convoluted data sources that entail a substantial degree of system internals' knowledge and source code visibility. For example, Jhumka and Leeke (2011), Abdelmoez et al (2004), Popic et al (2005), Cortellessa and Grassi (2007), and Voas (1997) require operation details, such as states and failure rates, for each system component, Hiller et al (2004), Hiller et al (2002a), Leeke and Jhumka (2010), and Michael and Jones (1997) leverage data obtained by instrumenting variables, while Tucek et al (2007) uses dynamic binary instrumentation.…”

WATCHER: in-situ failure diagnosis

“…Software contains latent bugs Tucek et al 2007]. Although software testing helps identify these bugs, the schedule pressure often causes vendors to release software without comprehensive testing.…”