Learning of Gaussian Processes in Distributed and Communication Limited Systems

Abstract

It is of fundamental importance to find algorithms obtaining optimal performance for learning of statistical models in distributed and communication limited systems. Aiming at characterizing the optimal strategies, we consider learning of Gaussian Processes (GPs) in distributed systems as a pivotal example. We first address a very basic problem: how many bits are required to estimate the inner-products of Gaussian vectors across distributed machines? Using information theoretic bounds, we obtain an optimal solution for the problem which is based on vector quantization. Two suboptimal and more practical schemes are also presented as substitute for the vector quantization scheme. In particular, it is shown that the performance of one of the practical schemes which is called per-symbol quantization is very close to the optimal one. Schemes provided for the inner-product calculations are incorporated into our proposed distributed learning methods for GPs. Experimental results show that with spending few bits per symbol in our communication scheme, our proposed methods outperform previous zero rate distributed GP learning schemes such as Bayesian Committee Model (BCM) and Product of experts (PoE).

Learning of Tree-Structured Gaussian Graphical Models on Distributed Data Under Communication Constraints

“…Some basic problems in machine learning such as classification, regression, hypothesis testing, etc. in distributed fashion are studied in [3], [33], [34]. Raginsky in [33] studied the classification and regression problem in distributed settings.…”

“…Many learning algorithms can be modified to run distributively at several machines to perform a learning task. There are many papers that propose distributed (parallel) version of various learning algorithms [1], [2], [3], [4]. However, some learning algorithms could not be efficiently parallelized on distributed data.…”

“…In general, three models can be considered for data in distributed settings: horizontal model, where the data are distributed across samples; vertical model, where data are distributed over dimensions; and hybrid model which is the combination of both horizontal and vertical models. Designing distributed learning systems for the horizontal model has been addressed in many papers [3], [5]. In [6] and [7], the vertical model is studied from information theoretic point of view.…”

Communication-aware Distributed Gaussian Process Regression Algorithms for Real-time Machine Learning

“…These methods speed up the training process and are able to scale to arbitrarily large datasets. Communication budget constraint is considered in [17] by reducing the dimensionality of transmitted data to approximate the whole dataset. Sparse approximation of full GPR is used in [17] to further relieve the communication overhead.…”

“…Communication budget constraint is considered in [17] by reducing the dimensionality of transmitted data to approximate the whole dataset. Sparse approximation of full GPR is used in [17] to further relieve the communication overhead. Notice that the server-client architecture requires each client being wellconnected with the server, and is not robust to the failure of the server.…”