Formal Verification of a Decision-Tree Ensemble Model and Detection of Its Violation Ranges

Sato, Naoto; Kuruma, Hironobu; Nakagawa, Yuichiroh; Ogawa, Hideto

doi:10.1587/transinf.2019edp7120

Cited by 13 publications

(11 citation statements)

References 33 publications

(42 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(Ranzato and Zanella 2020) presents an abstract interpretation method such that operations are conducted on the abstract inputs of the leaf nodes between trees. In (Sato et al 2019), the decision trees that compose the DTEM are encoded to a formula, and the formula is verified by using a satisfiability modulo theories (SMT) solver. considers the partitioning the input domain of decision trees into disjoint sets, exploring all feasible path combinations in the tree ensemble, and then deriving output tuples from leaves.…”

Section: Additional Experimental Resultsmentioning

confidence: 99%

“…There are a number of recent proposals (Kantchelian, Tygar, and Joseph 2016b;Ranzato and Zanella 2020;Sato et al 2019;Einziger et al 2019) on analysing the robustness of tree ensembles, with some of them converting the problem into a constraint solving problem so that an SMT solver can be used. Different from our goal of embedding and synthesis of knowledge, they are to determine the existence of adversarial examples.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Embedding and Extraction of Knowledge in Tree Ensemble Classifiers

Huang,

Zhao,

Huang

2020

Preprint

View full text Add to dashboard Cite

This paper studies the embedding and synthesis of knowledge in tree ensemble classifiers. We focus on knowledge expressible with a generic form of Boolean formulas, and show that a typical security attack, i.e., backdoor attack, is expressible with this knowledge expression. For the embedding, it is required to be preservative (i.e., the original performance of the classifier is preserved), verifiable (i.e., the knowledge can be attested), and stealthy (i.e., the embedding cannot be easily detected). To facilitate this, we propose two novel, and very effective, embedding algorithms, one of which is for blackbox setting and the other for white-box setting. The embedding can be done in PTIME. Beyond the embedding, we develop an algorithm to synthesise the embedded knowledge, by reducing the problem to be solvable with an SMT (satisfiability modulo theories) solver. While this novel algorithm can successfully synthesise knowledge, the reduction leads to an NP computation. Therefore, if applying embedding as security attack and synthesis as defence, our results suggest a complexity gap (P vs. NP) between security attack and security defence when working with machine learning models. We apply our algorithms to a diverse set of datasets to validate our conclusion empirically.

show abstract

Section: Additional Experimental Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Embedding and Extraction of Knowledge in Tree Ensemble Classifiers

Huang,

Zhao,

Huang

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…The experimental evaluation compares the latter approach to that of Kantchelian et al [91] and shows that it gives tight lower bounds with a speed-up of over 3000 times in one case (i.e., an ensemble with 300 trees trained on the HIGGS dataset [31]). [136] proposed an SMT-based approach for safety verification of random forests and gradient boosted decision trees. Specifically, their approach focuses on finding inputs that lead to a violation of a given output property.…”

Section: Formal Methods For Decision Tree Ensemblesmentioning

confidence: 99%

A Review of Formal Methods applied to Machine Learning

Urban¹,

Miné²

2021

Preprint

View full text Add to dashboard Cite

We review state-of-the-art formal methods applied to the emerging field of the verification of machine learning systems. Formal methods can provide rigorous correctness guarantees on hardware and software systems. Thanks to the availability of mature tools, their use is well established in the industry, and in particular to check safety-critical applications as they undergo a stringent certification process. As machine learning is becoming more popular, machine-learned components are now considered for inclusion in critical systems. This raises the question of their safety and their verification. Yet, established formal methods are limited to classic, i.e. non machine-learned software. Applying formal methods to verify systems that include machine learning has only been considered recently and poses novel challenges in soundness, precision, and scalability.We first recall established formal methods and their current use in an exemplar safety-critical field, avionic software, with a focus on abstract interpretation based techniques as they provide a high level of scalability. This provides a golden standard and sets high expectations for machine learning verification. We then provide a comprehensive and detailed review of the formal methods developed so far for machine learning, highlighting their strengths and limitations. The large majority of them verify trained neural networks and employ either SMT, optimization, or abstract interpretation techniques. We also discuss methods for support vector machines and decision tree ensembles, as well as methods targeting training and data preparation, which are critical but often neglected aspects of machine learning. Finally, we offer perspectives for future research directions towards the formal verification of machine learning systems.

show abstract

“…Michaelis et al [182] discuss a range of additional tools that serve similar verification purposes for deep learning models. Of course, similar formal verification techniques have also been proposed for other classes of machine learning models, including, in particular, various kinds of tree ensembles [207,208,209]. Several of these formal verification methods can also be used to iteratively "repair" a model that does not (yet) satisfy the specified constraints until it does [205].…”

Section: Model Evaluation Just Like a Large And Representativementioning

confidence: 99%

Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Technical Challenges and Solutions

Petersen,

Potdevin,

Mohammadi

et al. 2021

Preprint

View full text Add to dashboard Cite

Machine learning is expected to fuel significant improvements in medical care. To ensure that fundamental principles such as beneficence, respect for human autonomy, prevention of harm, justice, privacy, and transparency are respected, medical machine learning applications must be developed responsibly. A large number of high-level declarations of ethical principles have been put forth for this purpose, but there is a severe lack of technical guidelines explicating the practical consequences for medical machine learning. Similarly, there is currently considerable uncertainty regarding the exact regulatory requirements placed upon medical machine learning systems. In this paper, we survey the technical challenges involved in creating medical machine learning systems responsibly and in conformity with existing regulations, as well as possible solutions to address these challenges. We begin by providing a brief overview of existing regulations affecting medical machine learning, showing that properties such as safety, robustness, reliability, privacy, security, transparency, explainability, and nondiscrimination are all demanded already by existing law and regulations -albeit, in many cases, to an uncertain degree. Next, we discuss the key technical obstacles to achieving these desirable properties, and important techniques to overcome those barriers in the medical context. Since most of the technical challenges are very young and new problems frequently emerge, the scientific discourse is rapidly evolving and has not yet converged on clear best-practice solutions. Nevertheless, we aim to illuminate the underlying technical challenges, possible ways for addressing them, and their respective merits and drawbacks. In particular, we notice that distribution shift, spurious correlations, model underspecification, and data scarcity represent severe challenges in the medical context (and others) that are very difficult to solve with classical black-box deep neural networks. Important measures that may help to address these challenges include the use of large and representative datasets and federated learning as a means to that end, the careful exploitation of domain knowledge wherever feasible, the use of inherently transparent models, comprehensive model testing and verification, as well as stakeholder inclusion.

show abstract

Formal Verification of a Decision-Tree Ensemble Model and Detection of Its Violation Ranges

Cited by 13 publications

References 33 publications

Embedding and Extraction of Knowledge in Tree Ensemble Classifiers

Embedding and Extraction of Knowledge in Tree Ensemble Classifiers

A Review of Formal Methods applied to Machine Learning

Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Technical Challenges and Solutions

Contact Info

Product

Resources

About