MODES: model-based optimization on distributed embedded systems

Abstract

The predictive performance of a machine learning model highly depends on the corresponding hyper-parameter setting. Hence, hyper-parameter tuning is often indispensable. Normally such tuning requires the dedicated machine learning model to be trained and evaluated on centralized data to obtain a performance estimate. However, in a distributed machine learning scenario, it is not always possible to collect all the data from all nodes due to privacy concerns or storage limitations. Moreover, if data has to be transferred through low bandwidth connections it reduces the time available for tuning. Model-Based Optimization (MBO) is one state-of-the-art method for tuning hyper-parameters but the application on distributed machine learning models or federated learning lacks research. This work proposes a framework $$\textit{MODES}$$ MODES that allows to deploy MBO on resource-constrained distributed embedded systems. Each node trains an individual model based on its local data. The goal is to optimize the combined prediction accuracy. The presented framework offers two optimization modes: (1) $$\textit{MODES}$$ MODES -B considers the whole ensemble as a single black box and optimizes the hyper-parameters of each individual model jointly, and (2) $$\textit{MODES}$$ MODES -I considers all models as clones of the same black box which allows it to efficiently parallelize the optimization in a distributed setting. We evaluate $$\textit{MODES}$$ MODES by conducting experiments on the optimization for the hyper-parameters of a random forest and a multi-layer perceptron. The experimental results demonstrate that, with an improvement in terms of mean accuracy ($$\textit{MODES}$$ MODES -B), run-time efficiency ($$\textit{MODES}$$ MODES -I), and statistical stability for both modes, $$\textit{MODES}$$ MODES outperforms the baseline, i.e., carry out tuning with MBO on each node individually with its local sub-data set.

Federated Ensembles: a literature review

“…These do not only discuss straightforward classification tasks, but also attempt to solve other problems. Other subjects discussed included learning or sharing various forms of information in a privacy preserving manner, such as; generating synthetic data which can be used for further analyses without fear of leaking privacy sensitive data (Fioretto and Van Hentenryck, 2019), sharing labels (Martin and Zhu, 2021), learning hyperparameters (Shi et al, 2021), and fulfilling a segmentation task (Fay et al, 2020). Furthermore, there are publications focused on providing protection against various attacks (Cao et al, 2021).…”

A Dynamic Multitarget Detection Algorithm in front of Vehicle Based on Embedded System and Internet of Things

“…Some scholars applied the full convolution network technology to the three-dimensional scanning data to combine the threedimensional point dimension with the two-dimensional grid as the feature extraction method by forming different 2D endto-end full convolution networks through the candidate regions and detect the vehicle target and frame and achieved good results. Scholars of vehicle technology proposed to design a feature convolution kernel library composed of multiple forms and color Gabor [11] to train and screen the optimal feature extraction convolution kernel group by replacing the low-level convolution kernel group of the original network, so as to improve the detection accuracy [12]. In the upgrading iteration from single image detection to multitarget image detection, the adaptive threshold strategy is added to reduce the missed alarm rate and false alarm rate and realize the target detection in complex traffic scenes [13].…”

Artificial neural network and dataset optimization for implementation of linear system models in resource‐constrained embedded systems

“…Evaluation of model‐based optimization (MBO) on distributed embedded system with two different datasets using two different machine‐learning algorithms has been achieved. However, they require high‐end graphic processing units (GPUs) (Shi et al, 2021). Recently, neural network implementation for resource‐constrained embedded systems (ARM Cortex‐M) has been reported, but it lacks the on‐board training capabilities.…”