Metamorphic relations via relaxations: an approach to obtain oracles for action-policy testing

Enişer, Hasan Ferit; Gros, Timo P.; Wüstholz, Valentin; Hoffmann, Jörg; Christakis, Maria

doi:10.1145/3533767.3534392

Cited by 7 publications

(7 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We evaluate our testing procedure by checking a total of 38 properties found in (Eniser, Wüstholz, and Christakis 2023).…”

Section: Results For Testing Proceduresmentioning

confidence: 99%

“…We introduced many inspection and transformation functions; we now provide a summary of the remaining ones (see (Eniser, Wüstholz, and Christakis 2023) for details). The arity inspection function returns the number of parameters of a given function.…”

Section: Results For Testing Proceduresmentioning

confidence: 99%

“…Tables 1 and 2 show the percentages of (unique) property violations for all models. The absolute numbers are included in (Eniser, Wüstholz, and Christakis 2023) but are summarized in Table 3.…”

Section: Results For Testing Proceduresmentioning

confidence: 99%

“…When increasing the beam size to 3, the running time increases slightly for all models (for instance, to 3.3s for DOBF for Java to Python and to 51.3s for STARCODER for Java to Python). We include the average running time for all models in (Eniser, Wüstholz, and Christakis 2023).…”

Section: Results For Testing Proceduresmentioning

confidence: 99%

“…In Figure 1, we use transformation functions addConditional and renameParam, and inspection functions numConditionals, numLoops, compiles, and retValues. In total, we added 7 transformation and 5 inspection functions to express our properties-see (Eniser, Wüstholz, and Christakis 2023).…”

Section: Testing Proceduresmentioning

confidence: 99%

See 4 more Smart Citations

Automatically Testing Functional Properties of Code Translation Models

Eniser,

Wüstholz,

Christakis

2024

AAAI

View full text Add to dashboard Cite

Large language models are becoming increasingly practical for translating code across programming languages, a process known as transpiling. Even though automated transpilation significantly boosts developer productivity, a key concern is whether the generated code is correct. Existing work initially used manually crafted test suites to test the translations of a small corpus of programs; these test suites were later automated. In contrast, we devise the first approach for automated, functional, property-based testing of code translation models. Our general, user-provided specifications about the transpiled code capture a range of properties, from purely syntactic to purely semantic ones. As shown by our experiments, this approach is very effective in detecting property violations in popular code translation models, and therefore, in evaluating model quality with respect to given properties. We also go a step further and explore the usage scenario where a user simply aims to obtain a correct translation of some code with respect to certain properties without necessarily being concerned about the overall quality of the model. To this purpose, we develop the first property-guided search procedure for code translation models, where a model is repeatedly queried with slightly different parameters to produce alternative and potentially more correct translations. Our results show that this search procedure helps to obtain significantly better code translations.

show abstract

“…We evaluate our testing procedure by checking a total of 38 properties found in (Eniser, Wüstholz, and Christakis 2023).…”

Section: Results For Testing Proceduresmentioning

confidence: 99%

Section: Results For Testing Proceduresmentioning

confidence: 99%

Section: Results For Testing Proceduresmentioning

confidence: 99%

Section: Results For Testing Proceduresmentioning

confidence: 99%

Section: Testing Proceduresmentioning

confidence: 99%

See 3 more Smart Citations

Automatically Testing Functional Properties of Code Translation Models

Eniser,

Wüstholz,

Christakis

2024

AAAI

View full text Add to dashboard Cite

show abstract

Mutation‐Guided Metamorphic Testing of Optimality in AI Planning

Mazouni,

Gotlieb,

Spieker

et al. 2024

Software Testing Verif & Rel

View full text Add to dashboard Cite

Autonomous systems such as space‐ or underwater‐exploration robots or elderly people assistance robots often include an artificial intelligence (AI) planner as a component. Starting from the initial state of a system, an AI planner automatically generates sequential plans to reach final states that satisfy user‐specified goals. Generating plans having a minimum number of intermediate steps or taking the least time to execute is usually strongly desired, as these plans exhibit minimal costs. Unfortunately, testing if an AI planner generates optimal plans is almost impossible because the expected cost of these plans is usually unknown. Based on mutation adequacy test suite selection, this article proposes a novel metamorphic testing framework for detecting the lack of optimality in AI planners. The general idea is to perform a systematic but non‐exhaustive state space exploration from the initial state and to select mutant‐adequate states to instantiate new planning tasks as follow‐up test cases. We then check a metamorphic relation between the automatically generated solutions of the AI planner for these new test cases and the cost of the initial plan. We implemented this metamorphic testing framework in a tool called MorphinPlan. Our experimental evaluation shows that MorphinPlan can detect non‐optimal behaviour in both mutated AI planners and off‐the‐shelf, configurable planners. It also shows that our proposed mutation adequacy test selection strategy outperforms three alternative test generation and selection strategies, including both random state selection and random walks through the state space in terms of mutation scores.

show abstract