Federated Learning via Attentive Margin of Semantic Feature Representations

Abstract

Vision systems mounted on home robots need to interact with unseen classes in changing environments. Robots have limited computational resources, labelled data and storage capability. These requirements pose some unique challenges: models should adapt without forgetting past knowledge in a data-and parameter-efficient way. We characterize the problem as few-shot (FS) online continual learning (OCL), where robotic agents learn from a non-repeated stream of few-shot data updating only a few model parameters. Additionally, such models experience variable conditions at test time, where objects may appear in different poses (e.g., horizontal or vertical) and environments (e.g., day or night). To improve robustness of CL agents, we propose RobOCLe, which; 1) constructs an enriched feature space computing high order statistical moments from the embedded features of samples; and 2) computes similarity between high order statistics of the samples on the enriched feature space, and predicts their class labels. We evaluate robustness of CL models to train/test augmentations in various cases. We show that different moments allow RobOCLe to capture different properties of deformations, providing higher robustness with no decrease of inference speed.

Federated Learning in Computer Vision

“…In FL, a first possible strategy is the correction of the client drift by computing client deviations using margins of prototypical representations learned on distributed data. These margins can be exploited to drive the federated optimization, via an attention mechanism [161], to address system and statistical heterogeneity. Prototypes can also be transmitted instead of model weights [203], to reduce communication cost, to allow clients to learn a more customized local model and to be more robust to gradient-based attacks [28], [274] since high-level statistic information (prototypes) are more privacy-compliant than raw features.…”

“…Additionally, the distribution of the data across different clients may be heterogeneous, which can lead to performance degradation if not addressed appropriately. Despite these challenges, recent works have demonstrated promising results in applying federated learning to computer vision tasks [24], [62], [161], [193], and ongoing research aims at improving the performance and scalability of FL in this domain (an overview of the distribution of the papers on the different tasks is shown in Fig. 4).…”

“…As for the other tasks distributed training solutions must be considered due to privacy and efficiency constraints. FedMargin [161] provides a distributed framework for both image classification and semantic segmentation. It estimates client deviations from the margin of class-conditional representations, and uses this information to drive the federated optimization by means of an attention mechanism.…”

“…Most studies on federated learning focus on its theoretical and communication aspects [2], [105], [128], [145]. Nevertheless, Federated Learning has recently attracted a wide interest when applied to computer vision tasks, ranging from image classification [26], [88], [270] to semantic segmentation [24], [62], [161], and object detection [96], [147], [200].…”