Testing Non-Functional Requirements: Lacking of Technologies or Researching Opportunities?

Abstract

Software testing aims to reveal failures due to the lack of conformity (defects) among functional and non-functional requirements and the implemented system. Thus, defects can be identified and fixed, improving software quality. However, despite several works emphasizing the importance of non-functional requirements (NFRs), there is an insufficient amount of software testing approaches dealing with them. The lack of NFR evaluation may be the cause of low-quality software that does not meet users need, influencing software project success. Goal: To organize a body of knowledge regarding NFRs and software testing approaches available in the technical literature and reveal the gaps between testable NFRs and software testing approaches. Method: To perform structured literature reviews to identify NFRs and software testing approaches dealing with testable NFRs. To combine both results, reveal research opportunities and organize a body of knowledge regarding NFRs and software testing approaches. Results: From 224 identified NFRs, 87 were described, and 47 software testing approaches observed. Only eight approaches are empirically evaluated. No testing approaches were identified for 11 testable NFRs. Furthermore, regarding the testing process, we did not observe any testing approach covering the test planning phase. Conclusion: Despite their importance, many testable NFRs seem not be tested due to the lack of appropriate software testing approaches yet. Also, the existing testing approaches do not cover all testing processes activities and, in general, lack empirical evidence about their feasibility and performance, making their use in software projects risky.