Datamorphic Testing: A Method for Testing Intelligent Applications

Zhu, Hong; Li, Dongmei; Bayley, Ian; Harrison, Rachel; Cuzzolin, Fabio

doi:10.1109/aitest.2019.00018

Cited by 19 publications

(31 citation statements)

References 7 publications

(9 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Uncertainty emerges from a noncomplete specification or from requirements that are not fully understood. Algorithmic instability is of particular importance in AI applications [1]. This is due to the randomness inherent in the algorithms used in the development of AI.…”

Section: Algorithmic Uncertaintymentioning

confidence: 99%

“…In [1], we see examples of typical AI requirements that illustrate the testing challenge. We may have an application designed to carry out facial recognition.…”

Section: Lack Of a Testable Specificationmentioning

confidence: 99%

“…One of the biggest drivers is the concept of incomprehensibility. A neural network by its nature is very difficult to understand logically [1]. This makes many traditional methods of measuring code coverage such as branch or decision coverage next to impossible to compute.…”

Section: Incomprehensibilitymentioning

confidence: 99%

“…It is actually very common for AI techniques to be utilized in scenarios where the target environment is expected to be dynamic [1]. Voice recognition, for instance, is a popular field for AI implementation.…”

Section: Adaptive Behaviormentioning

confidence: 99%

“…The challenges that the model has in adjusting itself in performing more accurately in recognizing gender images [16] has been described clearly. [1] Focused on the issues faced in terms of testing facial recognition in AI systems. Data privacy is another major challenge in AI methodologies because of predictive analysis.…”

Section: Related Workmentioning

confidence: 99%

See 4 more Smart Citations

Software Testing: Issues and Challenges of Artificial Intelligence & Machine Learning

Sugali¹,

Sprunger²,

Inukollu³

2021

IJAIA

View full text Add to dashboard Cite

The history of Artificial Intelligence and Machine Learning dates back to 1950’s. In recent years, there has been an increase in popularity for applications that implement AI and ML technology. As with traditional development, software testing is a critical component of an efficient AI/ML application. However, the approach to development methodology used in AI/ML varies significantly from traditional development. Owing to these variations, numerous software testing challenges occur. This paper aims to recognize and to explain some of the biggest challenges that software testers face in dealing with AI/ML applications. For future research, this study has key implications. Each of the challenges outlined in this paper is ideal for further investigation and has great potential to shed light on the way to more productive software testing strategies and methodologies that can be applied to AI/ML applications.

show abstract

Section: Algorithmic Uncertaintymentioning

confidence: 99%

“…In [1], we see examples of typical AI requirements that illustrate the testing challenge. We may have an application designed to carry out facial recognition.…”

Section: Lack Of a Testable Specificationmentioning

confidence: 99%

Section: Incomprehensibilitymentioning

confidence: 99%

Section: Adaptive Behaviormentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Software Testing: Issues and Challenges of Artificial Intelligence & Machine Learning

Sugali¹,

Sprunger²,

Inukollu³

2021

IJAIA

View full text Add to dashboard Cite

show abstract

Supporting maintenance and testing for AI functions of mobile apps based on user reviews: An empirical study on plant identification apps

Tao

Guo

Zhang

et al. 2022

J Software Evolu Process

View full text Add to dashboard Cite

Despite the tremendous development of artificial intelligence (AI)‐based mobile apps, they suffer from quality issues. Data‐driven AI software poses challenges for maintenance and quality assurance. Metamorphic testing has been successfully adopted to AI software. However, most previous studies require testers to manually identify metamorphic relations in an ad hoc and arbitrary manner, thereby encountering difficulties in reflecting real‐world usage scenarios. Previous work showed that information available in user reviews is effective for maintenance and testing tasks. Yet, there is a lack of studies leveraging reviews to facilitate AI function maintenance and testing activities. This paper proposes METUR, a novel approach to supporting maintenance and testing for AI functions based on reviews. Firstly, METUR automatically classifies reviews that can be exploited for supporting AI function maintenance and evolution activities. Then, it identifies test contexts from reviews in the usage scenario category. METUR instantiates the metamorphic relation pattern for deriving concrete metamorphic relations based on test contexts. The follow‐up test dataset is constructed for conducting metamorphic testing. Empirical studies on plant identification apps indicate that METUR effectively categorizes reviews that are related to AI functions. METUR is feasible and effective in detecting inconsistent behaviors by using the metamorphic relations constructed based on reviews.

show abstract

Identifying metamorphic relations: A data mutation directed approach

Sun,

Jin,

et al. 2023

Softw Pract Exp

View full text Add to dashboard Cite

SummaryMetamorphic testing (MT) is an effective technique to alleviate the test oracle problem. The principle of MT is to detect failures by checking whether some necessary properties, commonly known as metamorphic relations (MRs), of software under test (SUT) hold among multiple executions of source and follow‐up test cases. Since both the generation of follow‐up test cases and test result verification depend on MRs, the identification of MRs plays a key role in MT, which is an important yet difficult task requiring deep domain knowledge of the SUT. Accordingly, techniques that can direct a tester to identify MRs effectively are desirable. In this paper, we propose MT, a data mutation directed approach to identifying MRs. MT guides a tester to identify MRs by providing a set of data mutation operators and template‐style mapping rules, which not only alleviates the difficulties faced in the process of MR identification but also improves the identification effectiveness. We have further developed a tool to implement the proposed approach and conducted an empirical study to evaluate the MR identification effectiveness of MT and the performance of MRs identified by MT with respect to fault detection capability and statement coverage. The empirical results show that MT is able to identify MRs for numeric programs effectively, and the identified MRs have high fault detection capability and statement coverage. The work presented in this paper advances the field of MT by providing a simple yet practical approach to the MR identification problem.

show abstract

Datamorphic Testing: A Method for Testing Intelligent Applications

Cited by 19 publications

References 7 publications

Software Testing: Issues and Challenges of Artificial Intelligence & Machine Learning

Software Testing: Issues and Challenges of Artificial Intelligence & Machine Learning

Supporting maintenance and testing for AI functions of mobile apps based on user reviews: An empirical study on plant identification apps

Identifying metamorphic relations: A data mutation directed approach

Contact Info

Product

Resources

About