Metamorphic Testing of Navigation Software: A Pilot Study with Google Maps

Brown, Joshua; Zhi, Zhou; Chow, Yang-Wai

doi:10.24251/hicss.2018.713

Cited by 29 publications

(14 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…A growing body of research has examined the concept of MT [6], [26]- [29], and proven it to be a very useful testing paradigm that effectively addresses the oracle problem. The increasing interest in MT is not only because of its ability to test software in the absence of an ideal oracle, but also because MT is based on a perspective not previously used by other testing techniques-as a result, it has been reported that MT can detect previously unknown faults in mature software systems such as the Siemens suite of programs [30], the GCC and LLVM compilers [3], [31], graphics shader compilers [32], various open source and commercial code obfuscators [33], and the Google Maps navigation system [34].…”

Section: Metamorphic Testing (Mt)mentioning

confidence: 99%

“…The route includes directions for the user to reach the destination. Not only is navigation software one of the most popular Internet applications, but it is actually the number one smart phone application, installed on more than 50%, globally [34]. Navigation systems are mission critical because errors in navigation may result in accidents, especially when such systems are used to guide selfdriving cars, delivery robots, or autonomous drones.…”

Section: Case Study Of Google Maps Naviga-tionmentioning

confidence: 99%

“…In this section, we report on using the Google Maps web application at maps.google.com to show how our method can help users and testers. Google Maps is by far the most popular mapping service [34].…”

Section: Case Study Of Google Maps Naviga-tionmentioning

confidence: 99%

See 2 more Smart Citations

Metamorphic Relations for Enhancing System Understanding and Use

Zhi

Sun

Chen

et al. 2020

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Modern information technology paradigms, such as online services and off-the-shelf products, often involve a wide variety of users with different or even conflicting objectives. Every software output may satisfy some users, but may also fail to satisfy others. Furthermore, users often do not know the internal working mechanisms of the systems. This situation is quite different from bespoke software, where developers and users usually know each other. This paper proposes an approach to help users to better understand the software that they use, and thereby more easily achieve their objectives-even when they do not fully understand how the system is implemented. Our approach borrows the concept of metamorphic relations from the field of metamorphic testing (MT), using it in an innovative way that extends beyond MT. We also propose a "symmetry" metamorphic relation pattern and a "change direction" metamorphic relation input pattern that can be used to derive multiple concrete metamorphic relations. Empirical studies reveal previously unknown failures in some of the most popular applications in the world, and show how our approach can help users to better understand and better use the systems. The empirical results provide strong evidence of the simplicity, applicability, and effectiveness of our methodology.

show abstract

Section: Metamorphic Testing (Mt)mentioning

confidence: 99%

Section: Case Study Of Google Maps Naviga-tionmentioning

confidence: 99%

See 1 more Smart Citation

Metamorphic Relations for Enhancing System Understanding and Use

Zhi

Sun

Chen

et al. 2020

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

show abstract

“…The unique perspective of MT (inspecting the relations among multiple executions -an area seldom explored by conventional testing methods) enabled the detection of previously unknown faults in a variety of real-world mature systems. Such examples include the detection of bugs in the GCC, LLVM, and other types of compilers and code obfuscators [19]- [22], in major search engines including Google, Bing, and Baidu [17], in the Web APIs of Spotify and YouTube [18], in the navigation system Google Maps [23] and, more recently, in self-driving cars' on-board computer software [14]. MT has also been applied to test NASA software [6], [24] and systematically adopted by Adobe Systems [10], [25].…”

Section: Introductionmentioning

confidence: 99%

Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors

Zhi

Tse

Witheridge

2021

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

We propose a robustness testing approach for software systems that process large amounts of data. Our method uses metamorphic relations to check software output for erroneous input in the absence of a tangible test oracle. We use this technique to test two major citation database systems: Scopus and the Web of Science. We report a surprising finding that the inclusion of hyphens in paper titles impedes citation counts, and that this is a result of the lack of robustness of the citation database systems in handling hyphenated paper titles. Our results are valid for the entire literature as well as for individual fields such as chemistry. We further find a strong and significant negative correlation between the journal impact factor (JIF) of IEEE Transactions on Software Engineering (TSE) and the percentage of hyphenated paper titles published in TSE. Similar results are found for ACM Transactions on Software Engineering and Methodology. A software engineering field-wide study reveals that the higher JIF-ranked journals are publishing a lower percentage of papers with hyphenated titles. Our results challenge the common belief that citation counts and JIFs are reliable measures of the impact of papers and journals, as they can be distorted simply by the presence of hyphens in paper titles.

show abstract

“…Metamorphic testing has been successfully used to alleviate the test oracle problem in different domains like compilers [16] [17], simulation software [18], web services [19], bioinformatics systems [20], navigation software [21], machine learning [22] [23], autonomous systems [24], cybersecurity [25] and search engines [26]. Testing games can be a great task as it can also result in multiple and complex scenarios.…”

Section: Introductionmentioning

confidence: 99%

Metamorphic Testing of an Artificially Intelligent Chess Game

2020

View full text Add to dashboard Cite

Artificially intelligent (AI) game software incorporates different algorithms to generate intelligent human-like responses to the users playing them. Testing AI game software poses great difficulty because of the complex possibilities that can result at a given point and analysis of said possibilities is a tedious task. Also during software development there are resource constraints due to which testing targets specific parts of the software. An AI game of Chess takes into consideration a large amount of possible outcomes at any given point before deciding a move. Therefore, testing it in its entirety is impractical. In this paper we propose a metamorphic testing approach for testing an AI Chess game i.e. a Chess engine's algorithm of determining and pruning out possible outcomes and ultimately deciding on a final outcome. For validating our approach, we have done error seeding on an open source Chess engine and tested it through our approach. The results for our proposed approach for testing an AI Chess game through metamorphic relations show that it is successful in revealing 71% of the total seeded faults. On comparison of our proposed approach with the existing technique of testing Chess engine i.e., perft function, we have come across situations in which our proposed approach reveals errors overlooked by the existing technique. In the future we aim to extend our approach towards other AI game software. INDEX TERMS artificial intelligence, Chess engine, Chess game, game testing, metamorphic testing, test oracle problem

show abstract

Metamorphic Testing of Navigation Software: A Pilot Study with Google Maps

Cited by 29 publications

References 9 publications

Metamorphic Relations for Enhancing System Understanding and Use

Metamorphic Relations for Enhancing System Understanding and Use

Metamorphic Robustness Testing: Exposing Hidden Defects in Citation Statistics and Journal Impact Factors

Metamorphic Testing of an Artificially Intelligent Chess Game

Contact Info

Product

Resources

About