An Exploratory Study on Handling Requirements and Acceptance Test Documentation in Industry

Abstract

With the emergence and spread of agile processes, the practices of writing and maintaining documentation have drastically changed in the last decade. In this work, we performed a qualitative study to explore the current practices for managing two related types of software documentation: requirements and acceptance tests. We interviewed twenty practitioners from seventeen business units in fifteen companies to investigate the companies' practices for writing, maintaining and linking requirements and acceptance test documentation. The study yields interesting and partially unexpected results. For example, we had expected that tests would be more extensively documented than requirements, while we found a strong linear correlation between the number of requirements and tests in our sample. We also found that technical people are usually not involved in the requirements engineering activities, which often results in misunderstood or underestimated requirements. Acceptance tests are written, in many cases, based on requirements that are not necessarily detailed enough. Also, acceptance tests are not regularly maintained, which occasionally results in confusing features and bugs. Abstract-With the emergence and spread of agile processes, the practices of writing and maintaining documentation have drastically changed in the last decade. In this work, we performed a qualitative study to explore the current practices for managing two related types of software documentation: requirements and acceptance tests. We interviewed twenty practitioners from seventeen business units in fifteen companies to investigate the companies' practices for writing, maintaining and linking requirements and acceptance test documentation. The study yields interesting and partially unexpected results. For example, we had expected that tests would be more extensively documented than requirements, while we found a strong linear correlation between the number of requirements and tests in our sample. We also found that technical people are usually not involved in the requirements engineering activities, which often results in misunderstood or underestimated requirements. Acceptance tests are written, in many cases, based on requirements that are not necessarily detailed enough. Also, acceptance tests are not regularly maintained, which occasionally results in confusing features and bugs.

Aligning Requirements and Acceptance Tests via Automatically Generated Guidance

“…Having non-aligned documents increases the risk of the late discovery of a mismatch between stakeholders' expectation and the actual software behaviour, which is one of the main reasons for the failure of many software projects [14]. In an earlier study, we found that requirements and acceptance tests are not aligned mostly due to (1) the manual effort needed for keeping the documents in a consistent state, (2) preferable verbal communication between requirements and test engineers and (3) the separation of the requirements and testing activities [9].…”

“…3) Notifying subscribed parties: We found in our previous study that testers are often not informed about changes in requirements [9]. Therefore, when a test fails, testers need a significant amount of time to realize that the cause is not a bug in the source code, but a change in the requirements.…”

“…Both researchers and practitioners are interested in binding requirements and testing activities more closely together [17], [5], [16], [10]. However, these activities, especially the documentation management, are still separated and performed by different people [9]. In order to keep the software documents aligned, many suggested traceability for change impact analysis using Information Retrieval (IR) and Natural language Processing (NLP) techniques.…”

“…Having weak communication between requirements engineers and test engineers often leads to confusing features and bugs [5]. For instance, if a requirement is extended by adding a new feature, but this change is not communicated to testers nor documented in the acceptance test, testers will report the actual feature to be a bug, regardless of the fact that the requirements document has changed [9]. Testers perform the steps specified in the acceptance tests and when the system does not behave the way it is specified in these tests, they report bugs without looking at the requirements documentation.…”

Keeping Evolving Requirements and Acceptance Tests Aligned with Automatically Generated Guidance

“…In practice, however, impacted artifacts too often are not kept aligned with changing requirements. To a significant extent, this is due to the additional effort required and to insufficient communication of requirement changes [1,2]. Losing the alignment between requirements and other documentation artifacts increases the risk of discovering mismatches between stakeholders' expectations and the actual software behavior only late, leading to unintended costs, delivery delays and unsatisfied customers.…”