Evaluating Bug Finders -- Test and Measurement of Static Code Analyzers

Delaitre, Aurélien; Stivalet, Bertrand; Fong, Elizabeth; Okun, Vadim

doi:10.1109/coufless.2015.10

Cited by 24 publications

(12 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We find 45 previously unknown bugs in OpenSSL implementation and 15 applications in Ubuntu which use SSL APIs, out of which 27 have been fixed. We share the lessons learned from bug detection and discussions with developers to 10 https://github.com/openssl/openssl/issues/6575 motivate more researchers and practicers to combat incorrect SSL API usages.…”

Section: Discussionmentioning

confidence: 99%

“…To automatically detect incorrect usages of SSL APIs in client programs, static analysis has long prevailed as one of the most promising techniques [10]. For example, He et al [11] design and implement SSLINT, a scalable static analysis tool to match a program dependence graph with a handcrafted, precise signature modeling the correct logic usage of SSL APIs.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

SSLDoc: Automatically Diagnosing Incorrect SSL API Usages in C Programs

Gu¹,

Wu²,

Li³

et al. 2019

International Conferences on Software Engineering and Knowledge Engineering

View full text Add to dashboard Cite

Secure Sockets Layer (SSL) and Transport Layer Security (TLS) protocols provide a reliable communication channel between applications over the Internet. Implementations of these protocols (e.g., OpenSSL and GnuTLS) publish wellformat documentation and examples online to guide the usage of SSL/TLS APIs. However, incorrect usages have caused many severe vulnerabilities (e.g., privilege escalation, denial of service, man-in-the-middle attack, etc.) in recent years. In this paper, we introduce SSLDoc to diagnose incorrect SSL API usages in real-world C programs automatically. The key insight behind SSLDoc is a constraint-directed static analysis technique powered by domain-specific usage patterns that we learn from real-world vulnerabilities and bug-fix-related patches. We have instantiated SSLDoc for OpenSSL APIs and applied it to large-scale opensource programs. SSLDoc found 45 previously unknown securitysensitive bugs in OpenSSL implementation and applications in Ubuntu. We created and submitted issues for all of them. Up to now, 35 have been confirmed by the corresponding development communities and 27 have been fixed in master branch.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

SSLDoc: Automatically Diagnosing Incorrect SSL API Usages in C Programs

Gu¹,

Wu²,

Li³

et al. 2019

International Conferences on Software Engineering and Knowledge Engineering

View full text Add to dashboard Cite

show abstract

“…A tool may find a weakness in one context, but not another context. Hence, test cases containing many occurrences of the same weakness type, surrounded by many code complexities [5], broadens the working range of a static analyzer, exploring more of its functionalities.…”

Section: Resultsmentioning

confidence: 99%

Impact of code complexity on software analysis

Oliveira¹,

Fong²,

Black³

2017

View full text Add to dashboard Cite

The Software Assurance Metrics and Tool Evaluation (SAMATE) team studied thousands of warnings from static analyzers. Tools have difficulty distinguishing between the absence of a weakness and the presence of a weakness that is buried in otherwise-irrelevant code elements. This paper presents classes of these code elements, which we call "code complexities."They have been present in software assurance as part of test cases generation strategy when evaluating static analyzers. Benefits of using code complexity include the development of coding guidelines, boosting diversification of test cases.

show abstract

“…Delaitre et al [7] evaluated 14 static analyzers. They establish three critical characteristics for vulnerability test cases and state that "Test cases with all three attributes are out of reach": 1.…”

Section: Insufficient Test Datamentioning

confidence: 99%

“…According to Nilson et al [29] and Delaitre et al [7], the existing databases are not sufficient for a comprehensive evaluation of bug finding techniques. However, that is not to say that there are no such databases.…”

Section: Vulnerability Databasesmentioning

confidence: 99%

EvilCoder

Pewny

Holz

2016

Proceedings of the 32nd Annual Conference on Computer Security Applications

View full text Add to dashboard Cite

The art of finding software vulnerabilities has been covered extensively in the literature and there is a huge body of work on this topic. In contrast, the intentional insertion of exploitable, security-critical bugs has received little (public) attention yet. Wanting more bugs seems to be counterproductive at first sight, but the comprehensive evaluation of bug-finding techniques suffers from a lack of ground truth and the scarcity of bugs. In this paper, we propose EvilCoder, a system to automatically find potentially vulnerable source code locations and modify the source code to be actually vulnerable. More specifically, we leverage automated program analysis techniques to find sensitive sinks which match typical bug patterns (e.g., a sensitive API function with a preceding sanity check), and try to find data-flow connections to usercontrolled sources. We then transform the source code such that exploitation becomes possible, for example by removing or modifying input sanitization or other types of security checks. Our tool is designed to randomly pick vulnerable locations and possible modifications, such that it can generate numerous different vulnerabilities on the same software corpus. We evaluated our tool on several open-source projects such as for example libpng and vsftpd, where we found between 22 and 158 unique connected source-sink pairs per project. This translates to hundreds of potentially vulnerable data-flow paths and hundreds of bugs we can insert. We hope to support future bug-finding techniques by supplying freshly generated, bug-ridden test corpora so that such techniques can (finally) be evaluated and compared in a comprehensive and statistically meaningful way.

show abstract

Evaluating Bug Finders -- Test and Measurement of Static Code Analyzers

Cited by 24 publications

References 5 publications

SSLDoc: Automatically Diagnosing Incorrect SSL API Usages in C Programs

SSLDoc: Automatically Diagnosing Incorrect SSL API Usages in C Programs

Impact of code complexity on software analysis

EvilCoder

Contact Info

Product

Resources

About