Counting Bugs is Harder Than You Think

Black, Paul E.

doi:10.1109/scam.2011.24

Cited by 10 publications

(9 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The challenge is in ensuring that the count is well-defined, repeatable, and reproducible. A simple concept like "software bug" can have hidden ambiguity that interferes with counting [40].…”

Section: Abatement Of Philosophical Quagmiresmentioning

confidence: 99%

“…Though bug counting is widely practiced and arguably "successful" as a measurement technique, there is no consensus definition of "bug" that is tight enough to ensure reproducible counts [40].…”

Section: Challengesmentioning

confidence: 99%

“…First, they do not consider the differences in consequences or the difficulty of exploitation for malicious purposes. Second, they are often ambiguous [40]. Finally, it is very difficult to enumerate all bugs in any reasonably large piece of software.…”

Section: Bugs (Correctness)mentioning

confidence: 99%

See 2 more Smart Citations

A Rational Foundation for Software Metrology

Flater¹,

Black²,

Fong³

et al. 2016

Self Cite

View full text Add to dashboard Cite

Section: Abatement Of Philosophical Quagmiresmentioning

confidence: 99%

Section: Challengesmentioning

confidence: 99%

See 1 more Smart Citation

A Rational Foundation for Software Metrology

Flater¹,

Black²,

Fong³

et al. 2016

Self Cite

View full text Add to dashboard Cite

“…Complementing our better understanding of where software faults lie, organizations such as NIST and MITRE have developed code repositories containing known faults [5,61,39]. This provides a benchmark by which analysis tools can be compared.…”

Section: Software Faults and Failuresmentioning

confidence: 99%

High Consequence Systems and Semantic Computing

Winter

Čukić

Khoshgoftaar

et al. 2013

Int. J. Semantic Computing

View full text Add to dashboard Cite

Socioeconomic needs combined with technological advances are creating a demand for an increasing number of systems for which high-assurance is an essential attribute. These system designs, which increasingly include semantic computing components, span a broad spectrum of applications. They are incredibly diverse and their complexity is growing. The conditions under which these systems operate are such that system faults will occur and must be comprehensively accounted for in the systems' designs.This article investigates the landscape of high-assurance systems, the challenges these systems face, and what must be done to address those challenges. Challenges include societal needs, scienti¯c frontiers, dynamics of technological evolution, and drivers of current research models.gathering Google Chairman Eric Schmidt said``We need to set out an agenda for innovation for our country as a whole" [14].In the book Engines of Innovation it is argued that research universities must step up and play a more central role in tackling``the world's biggest problems" [63]. The core of such innovation often rests on the successful leveraging of computer and information technology within the context of a larger system. We postulate that, in the years to come, semantic computing will provide an essential fulcrum in many such systems. Independent of the underlying technologies employed, the implications of technological dependence are enormous: In their 2010 report, the President's Council of Advisors on Science and Technology (PCAST) stated that Networking and Information Technology (NIT)``underpins our national prosperity, health, and security". It should not be surprising then that the new millennium has seen a distinct shift of academic missions in the direction of interdisciplinary and applied use of computer technology in a shift which is often referred to as translational science.It is the context of leveraging computer technology to solve``the world's biggest problems" that has led innovators to envision and develop a wide range of highconsequence systems. A classical de¯nition of a high-consequence system is a system in which a high``cost" (or consequence) is associated with failure. Here the cost metric is not limited to the monetary axis, but can range across any axis of social value such as human life, or national security. From this de¯nition, it can be argued that the terms``world's biggest problems" and``high-consequence problems" are practically synonymous. The corollary to this argument is that one can then expect solutions to the world's biggest problems to be realized though innovative highconsequence systems. It is worth noting that these system designs are extremely diverse: they include autonomous decentralized control systems such as the system used to control Japan's bullet train, marine renewable energy systems such as those being developed o® the Florida coast, medical systems permitting the sharing of clinical guidelines and synthesis of clinical work°ows and protocols, supervisory control and acquisition (...

show abstract

“…The comparison of results across tools, and for accuracy against the actual number of weaknesses, is performed by human analysts. These comparisons have proved to be intractable [1], and illustrate the need for precision and This work is sponsored by DHS contract FA8750-12-C-0277 automation, of the Juliet Test Suite variety, for real-world comparisons to be feasible. This paper reports on our current project, supported by the Department of Homeland Security (DHS), to provide such precision and automation for real-world comparisons using formal methods technology that is currently available.…”

Section: Introductionmentioning

confidence: 99%

A gold standard for assessing the coverage of static analyzers

Bush

2013

2013 IEEE International Conference on Technologies for Homeland Security (HST)

View full text Add to dashboard Cite

Static analyzers can be used at compile time to automatically discover locations in programs that are vulnerable to malicious exploits. It is difficult, however, to compare the results of one analyzer to another, or to assess the coverage of any one analyzer against the actual number of vulnerabilities present in a program, because there is no analyzer-independent metric of vulnerabilities for software. We are currently using our CodeHawk abstract interpretation technology to produce such a formal metric for 6 C programs from NIST's SATE competition. A single mathematical definition of undefined memory access according to the C semantics covers 27 of the MITRE Common Weakness Enumeration categories, and allows us to identify a set of program locations where proof obligations guaranteeing memory-safety cannot be discharged. These locations comprise a canonical enumeration of the weaknesses against which to measure static analyzers, including true positives (correctly identified locations), false positives (identified locations that can be proved safe), and false negatives (unsafe locations not identified).

show abstract

Counting Bugs is Harder Than You Think

Cited by 10 publications

References 16 publications

A Rational Foundation for Software Metrology

A Rational Foundation for Software Metrology

High Consequence Systems and Semantic Computing

A gold standard for assessing the coverage of static analyzers

Contact Info

Product

Resources

About