Evaluating capture and replay and model-based performance testing tools

Abstract

Context] A variety of testing tools have been developed to support and automate software performance testing activities. These tools may use di↵erent techniques, such as Model-Based Testing (MBT) or Capture and Replay (CR).[Goal] For software companies, it is important to evaluate such tools w.r.t. the e↵ort required for creating test artifacts using them; despite its importance, there are few empirical studies comparing performance testing tools, specially tools developed with di↵erent approaches.[Method] We are conducting experimental studies to provide evidence about the required e↵ort to use CR-based tools and MBT tools. In this paper, we present our first results, evaluating the e↵ort (time spent) when using LoadRunner and Visual Studio CRbased tools, and the PLeTsPerf MBT tool to create performance test scripts and scenarios to test Web applications, in the context of a collaboration project between Software Engineering Research Center at PUCRS and a technological laboratory of a global IT company.[Results] Our results indicate that, for simple testing tasks, the e↵ort of using a CR-based tool was lower than using an MBT tool, but as the testing complexity increases tasks, the advantage of using MBT grows significantly.[Conclusions] To conclude, we discuss the lessons we learned from the design, operation, and analysis of our empirical experiment.

PLeTsPerf - A Model-Based Performance Testing Tool

“…An important feature is the capability to generate the model based on scripts, i.e., the reverse engineering of our MBT approach; Lack of a standard performance language/script: One of the issues that motivated our research is the learning curve to replace or upgrade a load generator technology. The previous experiment's results [14] point that reuse of this drawback could be handled if the load generators use the same script or technique. For instance, an alternative to avoid this issue could be use a graphical notation, such as a Domain-Specific Languages (DSL) [16]…”

“…To better investigate these results we set up, in collaboration with the TDL, other experimental study. In this study, we compare the use of our tool, to generate scripts and scenarios, with two Capture & Replay tools: HP LoadRunner and MS Visual Studio [14].…”

Automatic Generation of Test Cases from UML Models

“…Regarding the information obtained, the majority of articles introduces the use of a single model for the generation of test cases and only six articles present the use of more than one model to generate the test cases. Four of them use two models [18], [19], [21], [25], and two use three models [26] [33]. Different types of models represent different views of the software, so that a greater number of models would make it possible to evaluate the software from their different views, thus providing greater coverage of the software.…”

“…Table 3 shows a comparison of TCGen regarding other tools that we found in our literature review. [18] Behavioural and state diagrams -abstract no EAST Test runner [18] --concrete -Condado [19] State transition diagram ---Mista [21] Petri nets and state transition diagrama Coverage criteria -yes TestMaster [22] State transition diagram -concrete no TaRGet [24] Use case diagram ---PLeTsPerf [25] Use case diagram and activity diagram concrete -SpecExplorer [28] State transition diagram -Concrete no LEIRIOS Smart Testing [32], [33] Class diagrams, objects diagrams, and state transition diagrams -Abstract -…”

Uma Proposta de Boas Práticas para Aplicação de Teste Baseado em Modelos em Métodos Ágeis

“…A partir dos resultados obtidos no mapeamento sistemático de literatura e das respostas coletadas nas entrevistas com os profissionais, identificamos as dificuldades e as vantagens da aplicação de MBT em métodoságeis que foram comuns nos dois estudos. Baseado nisso, e na experiência que possuímos, por integrarmos um grupo de pesquisa que trabalha faz dez anos com MBT [20], [8], [19], [18], [4], [5], analisamos estes pontos afim de propor um conjunto de boas práticas na implantação de MBT em times de desenvolvimentoágil. Por fim, realizamos uma survey com um conjunto de 30 profissionais que atuam em equipeságeis de desenvolvimento de sistemas para verificar se o conjunto de boas práticas sugeridas são coerentes e se elas podem auxiliar no processo de adoção de teste baseado em modelos por equipeságeis.…”