Cost Study of Test Automation Over Documentation for Microservices

Abstract

There are many development test frameworks targeted specifically for developers; e.g. Pytest [1], TestNG [2]. Additionally, there is a culture around developers where code is often preferred over documentation, e.g, Cucumber [3], Robot [4]. we define the category of test automations which provides readability as Test Over Documentation, and identify the common characteristics of it. Readability in the tests has many benefits, like helping improve code quality, however, adding readability to the test code affects the original structure of the test code, adding more cost to test development if the test code is poorly developed. Test automation is expensive, test over documentation can be even more expensive. The cost problem of building and maintaining test automation is one of the biggest problems in the arena of test automation [5], thus it is the same for Test Over Documentation. Test Over Documentation has many subcategories, each add readability to the tests in a different way, thus the cost is different. In this paper, we study the cost problem of Test Over Documentation, evaluate the cost of each subcategory for testing Microservices [6], hoping to give the testers some reference for tooling selection. Additionally, we propose a new design pattern named Element Pattern to help write Test Over Documentation with minimum cost and best readability. We also investigate if automatic test generation using model based testing [7] can help reduce cost. The experiment results compare different Test Over Documentation approaches against our proposed benchmark metrics.