A methodology for testing spreadsheets

Abstract

Spreadsheet languages, which include commercial spreadsheets and various research systems, have had a substantial impact on end-user computing. Research shows, however, that spreadsheets often contain faults; thus, we would like to provide at least some of the benefits of formal testing methodologies to the creators of spreadsheets. This article presents a testing methodology that adapts data flow adequacy criteria and coverage monitoring to the task of testing spreadsheets. To accommodate the evaluation model used with spreadsheets, and the interactive process by which they are created, our methodology is incremental. To accommodate the users of spreadsheet languages, we provide an interface to our methodology that does not require an understanding of testing theory. We have implemented our testing methodology in the context of the Forms/3 visual spreadsheet language. We report on the methodology, its time and space costs, and the mapping from the testing strategy to the user interface. In an empirical study, we found that test suites created according to our methodology detected, on average, 81% of the faults in a set of faulty spreadsheets, significantly outperforming randomly generated test suites.

Using assertions to help end-user programmers create dependable web macros

“…In the first loop (lines 2-20) each action node is associated with all its reachable clipboard elements and their types. The algorithm traverses a macro's WMG from top to bottom (lines [4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19], adding to each node the clipboard end if 31: end while variables and types from the previous node (union in line 6), identifying actions that set either a clipboard element (e.g., define action, clip action) or the type of a clipboard variable (e.g., filter action), and incorporating this information into the current action node. The first loop ends when no changes are observed on the collected clipboard type information across all action nodes, which is implemented through the noChangeInTypes construct.…”

“…The notion of using software engineering methodologies to help end users program more dependably in non-traditional language paradigms is not new in the software engineering research literature; in particular, it has been employed successfully in relation to the spreadsheet language paradigm (e.g., [6,19]). The notion of using assertions to enhance software dependability is also not new in the context of software engineering (e.g., [12,18]), and prior research has also considered automatic assertion generation in the context of professional programming [14].…”

AutoTest: A Tool for Automatic Test Case Generation in Spreadsheets

“…This aspect makes a strong case in favor of systematically building test suites, based on some coverage criterion, that can be run in as little time as possible, resulting in thorough testing of the program. The "What You See Is What You Test" (WYSIWYT) methodology for testing spreadsheets [21] allows users to test their spreadsheets to achieve DU adequacy. WYSIWYT has been developed for the Forms/3 spreadsheet language [7] and gives the user feedback of the overall level of testedness of the spreadsheet by means of a progress bar.…”

Integrating automated test generation into the WYSIWYT spreadsheet testing methodology

“…Thus, we have been developing a testing methodology for spreadsheets termed the "What You See Is What You Test" (WYSIWYT) methodology [43,44,45]. The WYSIWYT methodology provides feedback about the "testedness" of cells in spreadsheets in a way that is incremental, responsive, and entirely visual.…”