Estimating the Cost and Benefit of Model-Based Testing: A Decision Support Procedure for the Application of Model-Based Testing in Industry

Mohacsi, Stefan; Felderer, Michael; Beer, Armin

doi:10.1109/seaa.2015.18

Cited by 14 publications

(8 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We carefully identified and addressed potential threats to validity at every step of this MLR study. We followed the guidelines for performing SLRs and benefited from our previous experience, e.g., (Doğan et al 2014;Garousi et al 2015;Zhi et al 2015;Häser et al 2014;Mohacsi et al 2015;Catal and Diri 2009;Tarhan and Giray 2017;Giray and Tuzun 2018). This sub-section discusses the potential threats to validity and the actions we took to minimize or mitigate them.…”

Section: Potential Threats To Validity Of the Mlrmentioning

confidence: 99%

Practical relevance of software engineering research: synthesizing the community’s voice

2020

View full text Add to dashboard Cite

Software engineering (SE) research should be relevant to industrial practice. There have been regular discussions in the SE community on this issue since the 1980's, led by pioneers such as Robert Glass. As we recently passed the milestone of "50 years of software engineering", some recent positive efforts have been made in this direction, e.g., establishing "industrial" tracks in several SE conferences. However, many researchers and practitioners believe that we, as a community, are still struggling with research relevance and utility. The goal of this paper is to synthesize the evidence and experience-based opinions shared on this topic so far in the SE community, and to encourage the community to further reflect and act on the research relevance. For this purpose, we have conducted a Multi-vocal Literature Review (MLR) of 54 systematically-selected sources (papers and non peer-reviewed articles). Instead of relying on and considering the individual opinions on research relevance, mentioned in each of the sources, the MLR aims to synthesize and provide the "holistic" view on the topic. The highlights of our MLR findings are as follows. The top three root causes of low relevance, discussed in the community, are: (1) Researchers having simplistic views (or wrong assumptions) about SE in practice; (2) Lack of connection with industry; and (3) Wrong identification of research problems. The top three suggestions for improving research relevance are: (1) Using appropriate research approaches such as action-research; (2) Choosing relevant (practical) research problems; and (3) Collaborating with industry. By synthesizing all the discussions on this important topic so far, this paper aims to encourage further discussions and actions in the community to increase our collective efforts to improve the research relevance. Furthermore, we raise the need for empirically-grounded and rigorous studies on the relevance problem in SE research, as carried out in other fields such as management science.

show abstract

Section: Potential Threats To Validity Of the Mlrmentioning

confidence: 99%

Practical relevance of software engineering research: synthesizing the community’s voice

2020

View full text Add to dashboard Cite

show abstract

“…Furthermore, a lack of cost-benefit analysis in most research papers, i.e., how much effort/time has to be spent to adopt/implement a testing technique in a project and how much real cost savings will be yielded as a result, is highlighted as a major limitation of research ideas by practitioners. The authors have had some initial success in conducting cost-benefit analysis of testing techniques in their joint IACs with partners, e.g., [8][9][10].…”

Section: Analysis Goal and Approachmentioning

confidence: 99%

Worlds Apart: Industrial and Academic Focus Areas in Software Testing

Garousi

Felderer

2017

IEEE Softw.

Self Cite

View full text Add to dashboard Cite

The global software industry and the software engineering (SE) academia are two large communities. However, unfortunately, the level of joint industry-academia collaborations (IAC) in SE is very low, compared to the amount of activities in each of the two communities [1]. This is especially the case for software testing, which is a hot topic in SE research, reflected by the large amount of testing-related papers at recent ICSE conferences, and practice, reflected by the large amount and popularity of industrial testing conferences. The lack of mutual perception between industry and academia hurts both sides: researchers have fewer insights on the problems that are important to practitioners, while practitioners fail to learn what researchers have already discovered that might be useful to them. The issue of IAC in SE has been an important topic since the early years of SE (in the 1960's). In an applied field such as SE, industrial impact and relevance [2] are of outmost importance. For example, there are projects such as the ACM SIGSOFT Impact project (www.sigsoft.org/impact) which have measured and analyzed the impact of SE research on practice. But unfortunately, honestly speaking, many SE practitioners and researchers and especially those in software testing are not talking to each other (adequately). Various reasons have been discussed by researchers and practitioners for such a lack of motivation for collaborations between industry and academia [1], such as: each side having different objectives, industrial problems lacking scientific novelty or challenges, and the low applicability and scalability problems of the solutions developed in academia [3]. Also, a recent survey of 3,000 employees in Microsoft showed that unfortunately many practitioners do not find many of the top-cited SE research papers relevant or useful to their everyday challenges [2]. For the SE research community to have a meaningful future, there is a critical need to better connect industry and academia. There are even indications that the gap between industry and academia is even bigger in software testing compared to other areas of SE [4].

show abstract

“…Therefore, we present a comparison of time required to execute test cases manually and using ContextDrive. Mohacsi et al [30] reported time required to develop different testing artifacts including manual test case execution time and automated test case execution. Authors reported that on average 3 4 hour are required to execute a test case manually in which about 100 individual data checks need to be performed.…”

Section: ) Time Comparison Of Executing Test Scenario In Contextdrive Vs Manual Testingmentioning

confidence: 99%

ContextDrive: Towards a Functional Scenario-Based Testing Framework for Context-Aware Applications

Mirza

Khan²,

Wagan

et al. 2021

IEEE Access

View full text Add to dashboard Cite

Context-aware applications are emerging applications in the modern era of computing. These applications can determine and adapt to situational context to provide better user experience. Testing these applications is not straightforward. Constantly changing nature of context makes testing context-aware application is a challenging task. To uncover a defect in context-aware application, a test engineer needs activity (sequence of actions) and context information (context data); this makes test case development a difficult task. Conventional test case development methodologies do not cater for context information. Besides, conventional applications have only one input source, but context-aware application must obtain data from many sources to infer the context. Yet another issue that these applications often face is the noisy data problem as input data collected from physical sensors could be noisy. Test adequacy criteria are used as test stoppage rule and define the quality of testing as well as for generating test suites. Test adequacy criteria is helpful to control the cost of testing as well as determining/establishing confidence in the software product quality. A number of test adequacy criteria exist for testing conventional applications, but the same is not true for context-aware applications. Defining test adequacy criteria and test coverage measures for context-aware applications warrants further research. Several techniques have been developed by researchers to generate and execute test cases for context-aware applications; however, end-to-end testing and result analysis of executed test cases still remains a grueling task for the test engineers. The aim of this study is to automate end-to-end functional testing, analysis of the generated test results as well as functional/requirement coverage assessment. Moreover, we also present a confidence assessment template for result analysis. Test engineers can use our proposed framework to assess the requirement coverage. Our proposed framework will reduce testing time, efforts and cost thus enabling test engineers to execute more testing cycles to attain higher degree of test coverage. INDEX TERMSAutomatic testing, Context awareness, End-to-end testing, Model based testing, Scenariobased testing, Software quality , Software testing.

show abstract

Estimating the Cost and Benefit of Model-Based Testing: A Decision Support Procedure for the Application of Model-Based Testing in Industry

Cited by 14 publications

References 21 publications

Practical relevance of software engineering research: synthesizing the community’s voice

Practical relevance of software engineering research: synthesizing the community’s voice

Worlds Apart: Industrial and Academic Focus Areas in Software Testing

ContextDrive: Towards a Functional Scenario-Based Testing Framework for Context-Aware Applications

Contact Info

Product

Resources

About