What Makes Agile Test Artifacts Useful?

Abstract

Background: The artifacts used in Agile software testing and the reasons why these artifacts are used are fairly well-understood. However, empirical research on how Agile test artifacts are eventually designed in practice and which quality factors make them useful for software testing remains sparse. Aims: Our objective is twofold. First, we identify current challenges in using test artifacts to understand why certain quality factors are considered good or bad. Second, we build an Activity-Based Artifact Quality Model that describes what Agile test artifacts should look like. Method: We conduct an industrial survey with 18 practitioners from 12 companies operating in seven different domains. Results: Our analysis reveals nine challenges and 16 factors describing the quality of six test artifacts from the perspective of Agile testers. Interestingly, we observed mostly challenges regarding language and traceability, which are well-known to occur in non-Agile projects. Conclusions: Although Agile software testing is becoming the norm, we still have little confidence about general do's and don'ts going beyond conventional wisdom. This study is the first to distill a list of quality factors deemed important to what can be considered as useful test artifacts. CCS CONCEPTS • Software and its engineering → Agile software development; • General and reference → Empirical studies.

Verifying Agile Black-Box Test Case Quality Measurements: Expert Review

“…Whereas Fischbach et al [23] identified 16 quality characteristics for six Agile test artifacts, they focused on the Agile test artifacts in general. They proposed only one quality characteristic of the unit TCs: the code coverage, which is insufficient to measure TCQ.…”

“…1 above, it is clear that most studies presented the TCQ from a traditional approach. Four studies [15], [23], [41], [49] tried to improve the testing quality in the ASD method. Nevertheless, the majority did not define sub-characteristics; only one study proposed characteristics, sub-characteristics, and metrics developed by Lai [15].…”

“…But reducing the TCs may not solve the testing problems; even with a bit of effort in testing to deliver exemplary programs, bugs are still prevalent in many projects [16], [20]- [22]. The root causes of these testing problems were traced caused by the poor quality of TCs [13], [15], [16], [20], [21], [23]. Hence, it is essential to get good quality TCs that have a high chance of exposing defects, more accurate results, the ability to increase the system's performance, lower cost, and meet the user requirements [18], [24]- [29].…”

Explorando a especificação de cenários de teste de segurança no desenvolvimento orientado por comportamento (Behavior Driven Development)

“…Contudo, os autores reforçam a necessidade de refinar os achados voltando aos participantes da pesquisa para validar na prática quão úteis são os novos atributos de qualidade e a lista de verificação baseada em perguntas para que possam ser usados em contextos BDD no mundo real e obter resultados confiáveis. O estudo de [8] buscou identificar os desafios atuais no uso de artefatos de teste para entender por que certos fatores de qualidade são considerados bons ou ruins. Além disso, elaboraram um modelo de qualidade de artefato baseado em atividade que descreve como os artefatos de teste ágeis devem ser.…”

Conversion of User Story Scenarios to Python-Based Selenium Source Code for Automated Testing

“…User stories can also be used as the basis for test cases in software testing processes [8]- [10]. Acceptance criteria assigned to user stories are used as test cases and merged into acceptance tests, which will then be used to check the system behavior compliance with the user stories [11].…”