Choosing the Right Test Automation Tool

Raulamo-Jurvanen, Päivi; Mäntylä, Martti; Garousi, Vahid

doi:10.1145/3084226.3084252

Cited by 32 publications

(10 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cost is an important factor, but not considered to be a characteristic of product quality 1 . Rather, costs are categorized as a tool external factor [21]. In our prior research on tool evaluations [22], we found that practitioner evaluations for a tool, in a survey, may be dispersed.…”

Section: Related Workmentioning

confidence: 95%

“…It is important to understand the experiences, both positive and negative, related to those habits. Practitioners seem to have common but not systematically applied consensus about important criteria for selecting software testing tools [20,21]. For example, costs, in general is one frequently mentioned factor for the adoption and use of software testing tools [14,27,5,20,1].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Applying Surveys and Interviews in Software Test Tool Evaluation

Raulamo-Jurvanen

Hosio

Mäntylä

2019

Product-Focused Software Process Improvement

Self Cite

View full text Add to dashboard Cite

Despite the multitude of available software testing tools, literature lists lack of right tools and costs as problems for adopting a tool. We conducted a case study to analyze how a group of practitioners, familiar with Robot Framework (an open source, generic test automation framework), evaluate the tool. We based the case and the unit of analysis on our academia-industry relations, i.e., availability. We conducted a survey (n=68) and interviews (n=6) using convenience sampling to develop a comprehensive view of the phenomena. The study reveals the importance of understanding the interconnection of different criteria and the potency of the context on those. Our results show that unconfirmed or unfocused opinions about criteria, e.g., about Costs or Programming Skills, can lead to misinterpretations or hamper strategic decisions if overlooking required technical competence. We conclude surveys can serve as a useful instrument for collecting empirical knowledge about tool evaluation, but experiential reasoning collected with a complementary method is required to develop into comprehensive understanding about it.

show abstract

Section: Related Workmentioning

confidence: 95%

Section: Related Workmentioning

confidence: 99%

Applying Surveys and Interviews in Software Test Tool Evaluation

Raulamo-Jurvanen

Hosio

Mäntylä

2019

Product-Focused Software Process Improvement

Self Cite

View full text Add to dashboard Cite

show abstract

“…In grey literature, test tool evaluations tend to propose and include tasks like live trials, proof-of-concepts and demos [45]. Such tasks require resources and competence, and are considered to bear the risk of wrong decisions [39].…”

Section: Software Test Tool Selectionmentioning

confidence: 99%

“…We constructed a survey questionnaire including questions about background information and 15 questions for evaluating criteria (on different selected tools of choice), see Table 1. We used the criteria for the survey from a set of characteristics considered important by practitioners in test tool selection [44,45] and resting on the ISO/IEC 25010 1 quality model. The criteria to be evaluated were: (1) Applicability (2) Compatibility (3) Configurability (4) Cost-Effectiveness (5) Costs (6) Cross-Platform Support (7) Easy to Deploy (8) Easy to Use (9) Expandability (10) The respondents were able to select one or more tools and evaluate the criteria of choice for each tool, one tool at a time.…”

Section: Opinion Surveysmentioning

confidence: 99%

Practitioner Evaluations on Software Testing Tools

Raulamo-Jurvanen

Hosio

Mäntylä

2019

Proceedings of the Evaluation and Assessment on Software Engineering

Self Cite

View full text Add to dashboard Cite

In software engineering practice, evaluating and selecting the software testing tools that best fit the project at hand is an important and challenging task. In scientific studies of software engineering, practitioner evaluations and beliefs have recently gained interest, and some studies suggest that practitioners find beliefs of peers more credible than empirical evidence. To study how software practitioners evaluate testing tools, we applied online opinion surveys (n=89). We analyzed the reliability of the opinions utilizing Krippendorff's alpha, intra-class correlation coefficient (ICC), and coefficients of variation (CV). Negative binomial regression was used to evaluate the effect of demographics. We find that opinions towards a specific tool can be conflicting. We show how increasing the number of respondents improves the reliability of the estimates measured with ICC. Our results indicate that on average, opinions from seven experts provide a moderate level of reliability. From demographics, we find that technical seniority leads to more negative evaluations. To improve the understanding, robustness, and impact of the findings, we need to conduct further studies by utilizing diverse sources and complementary methods.

show abstract

“…Raulamo-Jurvanen et al [19] performed the only Grey Literature Review we have known in SE, aiming to analyze how software practitioners address the practical problem of choosing the right test automation tool. The data derived from experiences and opinions were present in most of the findings.…”

Section: Grey Literature In Sementioning

confidence: 99%

The Use of Grey Literature Review as Evidence for Software Engineering

Kamei¹,

Soares²,

Pinto³

2019

Anais Estendidos Do X Congresso Brasileiro De Software: Teoria E Prática (CBSoft Estendido 2019)

View full text Add to dashboard Cite

Context: In the last years, the Grey Literature (GL) had a notable increase in interest in diverse areas of knowledge. In Software Engineering (SE), with the advance of social media, the SE practitioners are heavy consumers of GL, opposite to traditional research papers, such as the Systematic Literature Review (SLR). Problem: Despite the increase of numbers of SLRs, some researchers claim the lack of connection to the practice of SE, due to the lack of evidence derived from SE practitioners. Goal: Introduce methods and techniques to support a Grey Literature Review (GLR) in SE, aiming to approximate the literature review to the practice of SE. Method: First, we will investigate how secondary studies use GL; Second, we will employ a survey with researchers of SLRs and SE practitioners; Third, we will analyze the use of GL according to the context; Fourth, we will review the GL source in SE; Finally, we will propose a guideline to help SE researchers and practitioners to conduct the GLR. Preliminary findings: We have initiated the tertiary study conducting the manual (4,875 studies) and automatic (6,279 studies) searches. We eliminated those that were not an article, and also those were not written in English, remaining 8,019 potential studies. Currently, we are applying the inclusion and exclusion criteria.

show abstract

Choosing the Right Test Automation Tool

Cited by 32 publications

References 18 publications

Applying Surveys and Interviews in Software Test Tool Evaluation

Applying Surveys and Interviews in Software Test Tool Evaluation

Practitioner Evaluations on Software Testing Tools

The Use of Grey Literature Review as Evidence for Software Engineering

Contact Info

Product

Resources

About