A Static Evaluation of Code Completion by Large Language Models

Ding, Hantian; Kumar, Vijay; Tian, Ye; Wang, Zijian; Kwiatkowski, Rafał; Li, Xiaopeng; Ramanathan, Murali; Ray, Baishakhi; Bhatia, Parminder; Sengupta, Sudipta

doi:10.18653/v1/2023.acl-industry.34

Cited by 2 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This underscores the generalizability of the importance of selectively retrieving API documentation when Code LLMs lack API specific knowledge. Additionally, scaling trends with model sizes (Kaplan et al, 2020;Wang et al, 2023a) are evident: average performance monotonically improves with model size in Table 3. Finally, DAG++ reveals that larger models require fewer retrievals for optimal performance, suggesting that they are more efficient at memorizing API syntax, even for low frequency APIs.…”

Section: Dag++ and Discussionmentioning

confidence: 90%

“…These hallucinations can propagate errors, creating a snowball effect (Zhang et al, 2023b). For e.g., a hallucinated API call can lead to hallucinated handling of its response in subsequent code segments, compounding the problem (Ding et al, 2023a). Such incorrect API usage can also introduce security vulnerabilities, like improper data handling, which may lead to attacks or data breaches (Pearce et al, 2021).…”

Section: Api Hallucinations and Cloudapibenchmentioning

confidence: 99%

See 1 more Smart Citation

API Documentation

Jain¹

2022

Learn API Testing

View full text Add to dashboard Cite

In this study, we address the issue of API hallucinations in various software engineering contexts. We introduce CloudAPIBench, a new benchmark designed to measure API hallucination occurrences. CloudAPIBench also provides annotations for frequencies of API occurrences in the public domain, allowing us to study API hallucinations at various frequency levels. Our findings reveal that Code LLMs struggle with low frequency APIs: for e.g., GPT-4o achieves only 38.58% valid low frequency API invocations. We demonstrate that Documentation Augmented Generation (DAG) significantly improves performance for low frequency APIs (increase to 47.94% with DAG) but negatively impacts high frequency APIs when using sub-optimal retrievers (a 39.02% absolute drop). To mitigate this, we propose to intelligently trigger DAG where we check against an API index or leverage Code LLMs' confidence scores to retrieve only when needed. We demonstrate that our proposed methods enhance the balance between low and high frequency API performance, resulting in more reliable API invocations (8.20% absolute improvement on CloudAPIBench for GPT-4o).

show abstract

Section: Dag++ and Discussionmentioning

confidence: 90%

Section: Api Hallucinations and Cloudapibenchmentioning

confidence: 99%