“…To this end, increasing the reliability of systems is becoming critically important. In spite of tremendous improvements in software engineering, testing and software reliability [34,29,30,22,31,24,33,16,27,14,13,12,7,26,36,38,40,21,15,11], many software bugs still escape testing and enter production. As others have noted [36], performing off-site analysis of production run failures at development sites has several limitations: 1) it is difficult to reproduce failures at the development site due to differences in the environment, 2) customers have privacy concerns over what information can be released for off-site diagnosis, and 3) the same bug may generate a different failure at multiple production sites; it is cumbersome for the development team to investigate every failure that occurs without any automated feedback regarding the root cause of the failure.…”