Training Binarized Neural Networks Using MIP and CP

Icarte, Rodrigo Toro; Illanes, León; Castro, Margarita P.; Ciré, André A.; McIlraith, Sheila A.; Beck, J. Christopher

doi:10.1007/978-3-030-30048-7_24

Cited by 16 publications

(34 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Section 4.2, we experimentally demonstrated that modeling ð instead of significantly improves the test performance of our MIP model. Our results suggest that similar works on training Binarized Neural Networks [11] using MIP models [25] might also benefit from similar explicit modeling of function ð.…”

Section: Interpretability Resultsmentioning

confidence: 66%

Training Experimentally Robust and Interpretable Binarized Regression Models Using Mixed-Integer Programming

Tule¹,

Lan²,

Say³

2021

Preprint

View full text Add to dashboard Cite

In this paper, we explore model-based approach to training robust and interpretable binarized regression models for multiclass classification tasks using Mixed-Integer Programming (MIP). Our MIP model balances the optimization of prediction margin and model size by using a weighted objective that: minimizes the total margin of incorrectly classified training instances, maximizes the total margin of correctly classified training instances, and maximizes the overall model regularization. We conduct two sets of experiments to test the classification accuracy of our MIP model over standard and corrupted versions of multiple classification datasets, respectively. In the first set of experiments, we show that our MIP model outperforms an equivalent Pseudo-Boolean Optimization (PBO) model and achieves competitive results to Logistic Regression (LR) and Gradient Descent (GD) in terms of classification accuracy over the standard datasets. In the second set of experiments, we show that our MIP model outperforms the other models (i.e., GD and LR) in terms of classification accuracy over majority of the corrupted datasets. Finally, we visually demonstrate the interpretability of our MIP model in terms of its learned parameters over the MNIST dataset. Overall, we show the effectiveness of training robust and interpretable binarized regression models using MIP.

show abstract

Section: Interpretability Resultsmentioning

confidence: 66%

Training Experimentally Robust and Interpretable Binarized Regression Models Using Mixed-Integer Programming

Tule¹,

Lan²,

Say³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Whereas a neural network is not typically trained through constrained optimization, we believe that our approach is more easily understood under such a mindset, which aligns with further work emerging from this community [8,31,15].…”

Section: Introductionmentioning

confidence: 60%

Training Thinner and Deeper Neural Networks: Jumpstart Regularization

Riera¹,

Rey²,

Serra³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural networks are more expressive when they have multiple layers. In turn, conventional training methods are only successful if the depth does not lead to numerical issues such as exploding or vanishing gradients, which occur less frequently when the layers are sufficiently wide. However, increasing width to attain greater depth entails the use of heavier computational resources and leads to overparameterized models. These subsequent issues have been partially addressed by model compression methods such as quantization and pruning, some of which relying on normalization-based regularization of the loss function to make the effect of most parameters negligible. In this work, we propose instead to use regularization for preventing neurons from dying or becoming linear, a technique which we denote as jumpstart regularization. In comparison to conventional training, we obtain neural networks that are thinner, deeper , and-most importantly-more parameter-efficient.

show abstract

“…Other researchers tried using information theory ideas in order to determine the optimal neural network size by having a tradeoff between complexity and a training error using second derivative information, which includes removing unimportant weights [ 184 ]. Researchers in [ 185 ] proposed a new method to train binarized neural networks at run-time; during forward propagation, this method greatly reduces the required memory size and replaces most operations with bit-wise operations [ 186 ]. However, binary weights were also proposed in [ 187 ], where researchers tried to replace the simple accumulations of several multiply-accumulate operations because multipliers took up most of the space and are considered power-hungry components when digital neural network is implemented.…”

Section: Deep Learning Solutions For Iot Data Compressionmentioning

confidence: 99%

The Deep Learning Solutions on Lossless Compression Methods for Alleviating Data Load on IoT Nodes in Smart Cities

Nasif

Othman

Sani

2021

Sensors

View full text Add to dashboard Cite

Networking is crucial for smart city projects nowadays, as it offers an environment where people and things are connected. This paper presents a chronology of factors on the development of smart cities, including IoT technologies as network infrastructure. Increasing IoT nodes leads to increasing data flow, which is a potential source of failure for IoT networks. The biggest challenge of IoT networks is that the IoT may have insufficient memory to handle all transaction data within the IoT network. We aim in this paper to propose a potential compression method for reducing IoT network data traffic. Therefore, we investigate various lossless compression algorithms, such as entropy or dictionary-based algorithms, and general compression methods to determine which algorithm or method adheres to the IoT specifications. Furthermore, this study conducts compression experiments using entropy (Huffman, Adaptive Huffman) and Dictionary (LZ77, LZ78) as well as five different types of datasets of the IoT data traffic. Though the above algorithms can alleviate the IoT data traffic, adaptive Huffman gave the best compression algorithm. Therefore, in this paper, we aim to propose a conceptual compression method for IoT data traffic by improving an adaptive Huffman based on deep learning concepts using weights, pruning, and pooling in the neural network. The proposed algorithm is believed to obtain a better compression ratio. Additionally, in this paper, we also discuss the challenges of applying the proposed algorithm to IoT data compression due to the limitations of IoT memory and IoT processor, which later it can be implemented in IoT networks.

show abstract

Training Binarized Neural Networks Using MIP and CP

Cited by 16 publications

References 13 publications

Training Experimentally Robust and Interpretable Binarized Regression Models Using Mixed-Integer Programming

Training Experimentally Robust and Interpretable Binarized Regression Models Using Mixed-Integer Programming

Training Thinner and Deeper Neural Networks: Jumpstart Regularization

The Deep Learning Solutions on Lossless Compression Methods for Alleviating Data Load on IoT Nodes in Smart Cities

Contact Info

Product

Resources

About