Automated metamorphic testing of variability analysis tools

Segura, Sergio; Durán, Amador; Sánchez, Ana María Sánchez; Berre, Daniel Le; Lonca, Emmanuel; Ruiz–Cortés, Antonio

doi:10.1002/stvr.1566

Cited by 17 publications

(15 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Metamorphic testing has been successfully applied to a number of testing domains including numerical programs [16], graph theory [17], service-oriented applications [14], or variability analysis tools [62]. In this article, it is applied to the automated analysis of FMs (see Section 5.1).…”

Section: Metamorphic Testingmentioning

confidence: 99%

“…Formally, let f m be the mandatory feature added to M and f p its parent feature. Then, the metamorphic relation between the products of M and those of M can be defined as shown below (see [62] for the formal definition of all metamorphic relationships in FMs).…”

Section: Metamorphic Testing On the Analysis Of Feature Modelsmentioning

confidence: 99%

“…The formalizations of OVM [51] and CUDF [69] are advanced (see Appendices A and B for details). We are currently working on the identification of the metamorphic relationships in both notations in order to be able to automatically generate test cases with BeTTy [62,63]. We are also considering the formalization of other FMlike notations, such as those including cardinalities [22], or those extending features with numerical information in the form of attributes and relationships among them [8,54,2].…”

Section: Specification Of More Variability Modeling Languagesmentioning

confidence: 99%

“…In this article, a CML specifying the semantics of an eclectic FM dialect known as basic feature model (BFM) [9] is presented, although other VMLs like OVM [51] or CUDF [69], which is used for package-based Linux distributions, can also be specified (see appendices A and B for an overview). During the development of FLAME, the verification and validation processes followed an approach in which formal verification were performed traditionally by manual theorem specification and proving, but validation were performed by integrating our experience on metamorphic testing of variability analysis tools [64,62], which has shown to be much more effective than manually-designed test cases, as described in [61]. To follow this automated, test-based validation approach, the specification of FLAME -developed in the standard, highly-expressive, well-known Z formal specification language [65,37]-was translated into Prolog [20] for testing purposes and 20,000 metamorphic random tests were automatically generated and executed.…”

Section: Introductionmentioning

confidence: 99%

“…In the case of FLAME, animating the Z specification in Prolog was very useful for detecting problems and inconsistencies by the manual execution of a small suite of tests, which were very helpful during the discussions among the authors about the semantics of some operations and provided immediate feedback. On the other hand, the animation was also systematically tested against 20,000 random test cases automatically generated using metamorphic testing techniques inspired by the previous work of some of the authors [64,62]. Metamorphic testing [15], exploits the relations between the inputs and outputs of a program to generate new followup test cases from existing test data (see Section 2.3 for an overview and Section 5.1 for details).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

FLAME: a formal framework for the automated analysis of software product lines validated by automated specification testing

et al. 2015

Self Cite

View full text Add to dashboard Cite

In a literature review on the last 20 years of automated analysis of feature models, the formalization of analysis operations was identified as the most relevant challenge in the field. This formalization could provide very valuable assets for tool developers such as a precise definition of the analysis operations and, what is more, a reference implementation, i.e. a trustworthy, not necessarily efficient implementation to compare different tools outputs. In this article, we present the FLAME framework as the result of facing this challenge. FLAME is a formal framework that can be used to formally specify not only feature models, but other variability modeling languages (VMLs) as well. This reusability is achieved by its two-layered architecture. The abstract foundation layer is the bottom layer in which all VMLindependent analysis operations and concepts are specified. On top of the foundation layer, a family of characteristic model layers-one for each VML to be formally specifiedcan be developed by redefining some abstract types and relations. The verification and validation of FLAME has fol- lowed a process in which formal verification has been performed traditionally by manual theorem proving, but validation has been performed by integrating our experience on metamorphic testing of variability analysis tools, something that has shown to be much more effective than manuallydesigned test cases. To follow this automated, test-based validation approach, the specification of FLAME, written in Z, was translated into Prolog and 20,000 random tests were automatically generated and executed. Tests results helped to discover some inconsistencies not only in the formal specification, but also in the previous informal definitions of the analysis operations and in current analysis tools. After this process, the Prolog implementation of FLAME is being used as a reference implementation for some tool developers, some analysis operations have been formally specified for the first time with more generic semantics, and more VMLs are being formally specified using FLAME.

show abstract

Section: Metamorphic Testingmentioning

confidence: 99%

Section: Metamorphic Testing On the Analysis Of Feature Modelsmentioning

confidence: 99%

Section: Specification Of More Variability Modeling Languagesmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

FLAME: a formal framework for the automated analysis of software product lines validated by automated specification testing

et al. 2015

Self Cite

View full text Add to dashboard Cite

show abstract

Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels

Kanewala

Bieman

Ben‐Hur

2015

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary Comprehensive, automated software testing requires an oracle to check whether the output produced by a test case matches the expected behaviour of the programme. But the challenges in creating suitable oracles limit the ability to perform automated testing in some programmes, and especially in scientific software. Metamorphic testing is a method for automating the testing process for programmes without test oracles. This technique operates by checking whether the programme behaves according to properties called metamorphic relations. A metamorphic relation describes the change in output when the input is changed in a prescribed way. Unfortunately, finding the metamorphic relations satisfied by a programme or function remains a labour‐intensive task, which is generally performed by a domain expert or a programmer. In this work, we propose a machine learning approach for predicting metamorphic relations that uses a graph‐based representation of a programme to represent control flow and data dependency information. In earlier work, we found that simple features derived from such graphs provide good performance. An analysis of the features used in this earlier work led us to explore the effectiveness of several representations of those graphs using the machine learning framework of graph kernels, which provide various ways of measuring similarity between graphs. Our results show that a graph kernel that evaluates the contribution of all paths in the graph has the best accuracy and that control flow information is more useful than data dependency information. The data used in this study are available for download at http://www.cs.colostate.edu/saxs/MRpred/functions.tar.gz to help researchers in further development of metamorphic relation prediction methods. Copyright © 2015 John Wiley & Sons, Ltd.

show abstract

Testing Chatbots Using Metamorphic Relations

Božić

Wotawa

2019

Testing Software and Systems

View full text Add to dashboard Cite

Modern-day demands for services often require an availability on a 24/7 basis as well as online accessibility around the globe. For this sake, personalized software systems, called chatbots, are applied. Chatbots offer services, goods or information in natural language. These programs respond to the user in real-time and offer an intuitive and simple interface to interact with. Advantages like these makes them increasingly popular. Chatbots can even act as substitutes for humans for specific purposes. Since the chatbot market is growing, chatbots might outperform and replace classical web applications in the future. For this reason, ensuring correct functionality of chatbots is of high and increasing importance. However, since different implementations and user behavior result in unpredictable results, the chatbot's output is difficult to predict and classify as well. In fact, testing of chatbots represents a challenge because of the unavailability of a test oracle. In this paper, we introduce a metamorphic testing approach for chatbots. In general, metamorphic testing can be applied to situations where no expected values are available. In addition, we discuss how to obtain test cases for chatbots, i.e. sequences of interactions with a chatbot, in an according manner. We demonstrate our approach using a hotel booking system and discuss first experimental results.

show abstract

Automated metamorphic testing of variability analysis tools

Cited by 17 publications

References 49 publications

FLAME: a formal framework for the automated analysis of software product lines validated by automated specification testing

FLAME: a formal framework for the automated analysis of software product lines validated by automated specification testing

Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels

Testing Chatbots Using Metamorphic Relations

Contact Info

Product

Resources

About