Adaptive Test Generation Using a Large Language Model

Schäfer, Max; Nadi, Sarah; Eghbali, Aryaz; Tip, Frank

doi:10.48550/arxiv.2302.06527

Cited by 5 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1) During our preliminary concept-distillation experiments, ChatGPT returns a wide range of concepts, including highly relevant, irrelevant, and sometimes duplicated ones, during the looped execution. As discussed in [5], [6], with more specific context information and good examples come improved semantic accuracy and more focused responses. Thus, we need to provide illustrative examples in the prompt to distil those highly relevant concepts while eliminating the rest.…”

Section: B Framework Overviewmentioning

confidence: 89%

“…Large language models (LLMs), such as GPT-3 [1], Codex [2], and ChatGPT [3] have made remarkable progress. Trained using an enormous amount of indiscriminate data from the entire internet, these LLMs embed knowledge from different domains, which are thus capable of answering questions, writing codes, drawing pictures, or translating languages across application areas [4]- [6]. In this paper, we aim to investigate if and how the knowledge of a specific application domain, e,g., scenario-based testing of autonomous vehicles, can be extracted to facilitate subsequent tasks, e.g.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain

Tang,

Da Costa,

Zhang

et al. 2023

2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC)

View full text Add to dashboard Cite

show abstract

Section: B Framework Overviewmentioning

confidence: 89%

Section: Introductionmentioning

confidence: 99%

Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain

Tang,

Da Costa,

Zhang

et al. 2023

2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC)

View full text Add to dashboard Cite

show abstract

“…Language models have been adapted to perform program fuzzing (Xia et al, 2023a;Deng et al, 2023), test generation (Schäfer et al, 2023), automated program repair (Xia et al, 2023b, and source-level algorithmic optimization Madaan et al (2023). The introduction of fill-in-the-middle capabilities is especially useful for software engineering use cases such as code completion, and has become common in recent code models such as InCoder (Fried et al, 2023), SantaCoder (Allal et al, 2023, StarCoder (Lozhkov et al, 2024), andCode Llama (Rozière et al, 2023).…”

Section: Related Workmentioning

confidence: 99%

CompilerGym: Robust, Performant Compiler Optimization Environments for AI Research

Cummins¹,

Wasti²,

Guo³

et al. 2022

2022 IEEE/ACM International Symposium on Code Generation and Optimization (CGO)

View full text Add to dashboard Cite

Large Language Models (LLMs) have demonstrated remarkable capabilities across a variety of software engineering and coding tasks. However, their application in the domain of code and compiler optimization remains underexplored. Training LLMs is resource-intensive, requiring substantial GPU hours and extensive data collection, which can be prohibitive. To address this gap, we introduce Meta Large Language Model Compiler (LLM Compiler), a suite of robust, openly available, pre-trained models specifically designed for code optimization tasks. Built on the foundation of Code Llama, LLM Compiler enhances the understanding of compiler intermediate representations (IRs), assembly language, and optimization techniques. The model has been trained on a vast corpus of 546 billion tokens of LLVM-IR and assembly code and has undergone instruction fine-tuning to interpret compiler behavior. LLM Compiler is released under a bespoke commercial license to allow wide reuse and is available in two sizes: 7 billion and 13 billion parameters. We also present fine-tuned versions of the model, demonstrating its enhanced capabilities in optimizing code size and disassembling from x86_64 and ARM assembly back into LLVM-IR. These achieve 77% of the optimising potential of an autotuning search, and 45% disassembly round trip (14% exact match). This release aims to provide a scalable, cost-effective foundation for further research and development in compiler optimization by both academic researchers and industry practitioners.

show abstract

“…There has been some research on using LLMs for white-box test generation. Schäfer et al [1] and Li et al [2] have explored ChatGPT in this regard, demonstrating its potential for generating effective white-box test cases. However, using LLMs for black-box testing has not been delved into much.…”

Section: Introductionmentioning

confidence: 99%

LatteArt: A Platform for Recording and Analyzing Exploratory Testing

Kirinuki¹,

Tajima²,

Tanno³

2023

2023 IEEE Conference on Software Testing, Verification and Validation (ICST)

View full text Add to dashboard Cite

In recent years, large language models (LLMs), such as ChatGPT, have been pivotal in advancing various artificial intelligence applications, including natural language processing and software engineering. A promising yet underexplored area is utilizing LLMs in software testing, particularly in black-box testing. This paper explores the test cases devised by ChatGPT in comparison to those created by human participants. In this study, ChatGPT (GPT-4) and four participants each created black-box test cases for three applications based on specifications written by the authors. The goal was to evaluate the real-world applicability of the proposed test cases, identify potential shortcomings, and comprehend how ChatGPT could enhance human testing strategies. ChatGPT can generate test cases that generally match or slightly surpass those created by human participants in terms of test viewpoint coverage. Additionally, our experiments demonstrated that when ChatGPT cooperates with humans, it can cover considerably more test viewpoints than each can achieve alone, suggesting that collaboration between humans and ChatGPT may be more effective than human pairs working together. Nevertheless, we noticed that the test cases generated by ChatGPT have certain issues that require addressing before use.

show abstract

Adaptive Test Generation Using a Large Language Model

Cited by 5 publications

References 0 publications

Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain

Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain

CompilerGym: Robust, Performant Compiler Optimization Environments for AI Research

LatteArt: A Platform for Recording and Analyzing Exploratory Testing

Contact Info

Product

Resources

About