Data debugging with continuous testing

Abstract

Today, systems rely as heavily on data as on the software that manipulates those data. Errors in these systems are incredibly costly, annually resulting in multi-billion dollar losses, and, on multiple occasions, in death. While software debugging and testing have received heavy research attention, less effort has been devoted to data debugging: discovering system errors caused by well-formed but incorrect data. In this paper, we propose continuous data testing: using otherwise-idle CPU cycles to run test queries, in the background, as a user or database administrator modifies a database. This technique notifies the user or administrator about a data bug as quickly as possible after that bug is introduced, leading to at least three benefits: (1) The bug is discovered quickly and can be fixed before it is likely to cause a problem. (2) The bug is discovered while the relevant change is fresh in the user's or administrator's mind, increasing the chance that the underlying cause of the bug, as opposed to only the discovered side-effect, is fixed. (3) When poor documentation or company policies contribute to bugs, discovering the bug quickly is likely to identify these contributing factors, facilitating updating documentation and policies to prevent similar bugs in the future. We describe the problem space and potential benefits of continuous data testing, our vision for the technique, challenges we encountered, and our prototype implementation for PostgreSQL.

Continuous software engineering: A roadmap and agenda

“…Similar to continuous integration, there are potential benefits to this (Muslu et al, 2013). Firstly, errors can be fixed quickly while the context is fresh in the developers' minds and before these errors lead to knock-on problems.…”

Making offline analyses continuous

“…Similarly, continuous testing [24,25,26] executes the tests available for a program as it is being developed. Meanwhile, continuous data testing applies the same ideas to data debugging: discovering system errors caused by well-formed but incorrect data [20]. While continuous testing is integrated into Eclipse [27] and IntelliJ [15], Solstice provides a generic framework for converting offline analyses into continuous ones, integrated into the IDE.…”

“…To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ESEC/FSE '13, August [18][19][20][21][22][23][24][25][26]2013, Saint Petersburg, Russia Copyright 2013 ACM 978-1-4503-2237-9/13/08 ...$15.00.…”

Continuous software engineering and beyond: trends and challenges

“…Similar to continuous integration, there are potential benefits to this [43]. Firstly, errors can be fixed quickly while the context is fresh in the developers' minds and before these errors lead to knock-on problems.…”