Automated Acceptance Tests as Software Requirements: An Experiment to Compare the Applicability of Fit Tables and Gherkin Language

Abstract

Abstract. It is estimated that 85% of the defects in the developed software are originated from ambiguous, incomplete and wishful thinking software requirements. Natural language is often used to write software requirements specifications as well as user requirements. However, natural language specifications can be confusing and hard to understand. Some agile methodologists consider that acceptance tests are more precise and accurate sources of information about the customer's needs than descriptions in natural language. Several studies have addressed the use of acceptance tests as software requirements specification. Therefore, none of the previous studies has performed experiments to compare the applicability of different acceptance testing techniques in order to support an organization in the selection of one technique over another. This paper addresses this problem reporting an experiment conducted with undergraduate students in Computer Science. This experiment compares the applicability of two acceptance testing techniques (Fit tables and Gherkin language) as software requirements specification. This research tries to answer three questions: (a) Which technique is the easiest to learn in order to specify acceptance test scenarios? (b) Which technique requires less effort to specify acceptance tests? (c) Which technique is the best one to communicate software requirements? The results show that there is no sufficient evidence to affirm that one technique is easier to specify test scenarios or better to communicate software requirements. Whereas, the comparison of effort in terms of time to specify acceptance testing shows that the mean time to specify test scenarios using Gherkin language is lower than Fit tables.

A systematic reuse process for automated acceptance tests: Construction and elementary evaluation

“…An experiment on comparing text-based AATs and FIT tables revealed that text-based AATs are easier to understand. These tests are developed in shorter duration as compared to FIT tables [31].…”

Towards Modelling Acceptance Tests as a Support for Software Measurement

“…Focusing on such state-of-practice, the most common strategies in agile testing are Test-Driven Development (TDD), Acceptance-Test-Driven Development (ATDD), and Behavior-Driven Development (BDD) [19]. Acceptance tests are expressed as code (e.g., using the Gherkin language [20]), creating an executable bridge between the user stories they are validating the implemented product. Solís and Wang advocate that ATDD is widely adopted in the industry because it im-proves software quality and productivity [21].…”

Impacts of Data Uniformity in the Reuse of Acceptance Test Glue Code

“…In the agile software development, acceptance tests are adopted to enable the communication and collaboration among the stakeholders [1,2,3]. Some agile developers, users and clients use acceptance testing as a way of specifying software requirements instead of using only common artifacts based on natural language [17,18]. This is an attempt to improve the quality of requirements because several problems can arise when requirements are written in natural language, for example, readers and writers can use the same word for different concepts, or express the same concept in completely different ways [18].…”