Addressing High Severity Faults in Web Application Testing

Abstract

Extreme resource constraints in web application development environments often lead to minimal formal testing. This lack of testing compounds the problem of low consumer retention. In this paper, we analyze an existing study of the consumer-perceived severities of 400 realworld web application faults. We provide concrete guidelines to increase the perceived return-on-investment of testing. We show that even coarse-grained, automated test cases can detect high severity faults. Developers can also mitigate consumer perceptions of fault severity by presenting faults using specific idioms that minimize disruptions to application interaction. Finally, we examine the trade offs between various user-session-based test suite reduction approaches for web applications and the severity of uncovered faults. Overall, we show that even modest testing approaches are able to provide significant predicted gains in consumer satisfaction by focusing on flagging and preventing high severity faults.

Automating regression testing using web-based application similarities

“…Despite this, the overall predictive power (F-value) of this feature was generally low, with the exception of Click where error keywords were able to perfectly predict actual faults. The main reason is that none of these real-world browser-based applications ever displayed a stack trace directly, and instead wrapped their visible errors in more human-friendly formats; this is in contrast to our experience with web-based application faults in general [12]. From a consumer perspective, it should be possible to design web-based applications that fail elegantly without stack traces or upsetting error messages [12] and yet still rely on a sophisticated tool such as Smart that can nevertheless detect such failures without relying on a search for any particular keywords in the document.…”

“…We follow previous work [11,12] in dividing fault severity (i.e., consumer perceptions of fault importance) into four levels: severe, medium, low and very low. These severity ratings correspond to varying levels of human actions based upon a fault seen; for example, severe faults occur when the user would either file a complaint or probably not return to the website.…”

“…-An additional experiment evaluating our automated comparator on the new real-world browser-based application benchmarks that includes an analysis of the severity of actual bugs our tool failed to flag. Relying on an automated model of fault severity from [11,12], we are able to quantitatively and qualitatively assign consumerperceived severity ratings to the faults uncovered and missed by our approach, with the expectation that missed faults will generally be non-severe.…”

Modeling consumer-perceived web application fault severities for testing

“…In [10] we explored how the model of fault severity in this work can be applied to software engineering practices, in an effort to focus on high-severity faults. Under the assumption that few to no resources are available for testing activities (given the extreme resource constraints of the web application development environment), [10] investigates the relationship between high severity faults and the visual presentation of errors and their causes.…”

“…Under the assumption that few to no resources are available for testing activities (given the extreme resource constraints of the web application development environment), [10] investigates the relationship between high severity faults and the visual presentation of errors and their causes. For example, even though developers may be unable to prevent all errors from manifesting in a user-visible fashion, they can often choose to display defects elegantly in the form of a popup or customized error message, as opposed to a stack trace; the former presentations are associated with lower consumerperceived severity.…”