Federated Self-Supervised Learning of Multisensor Representations for Embedded Intelligence

Abstract

Smartphones, wearables, and Internet of Things (IoT) devices produce a wealth of data that cannot be accumulated in a centralized repository for learning supervised models due to privacy, bandwidth limitations, and the prohibitive cost of annotations. Federated learning provides a compelling framework for learning models from decentralized data, but conventionally, it assumes the availability of labeled samples, whereas on-device data are generally either unlabeled or cannot be annotated readily through user interaction. To address these issues, we propose a self-supervised approach termed scalogramsignal correspondence learning based on wavelet transform to learn useful representations from unlabeled sensor inputs, such as electroencephalography, blood volume pulse, accelerometer, and WiFi channel state information. Our auxiliary task requires a deep temporal neural network to determine if a given pair of a signal and its complementary viewpoint (i.e., a scalogram generated with a wavelet transform) align with each other or not through optimizing a contrastive objective. We extensively assess the quality of learned features with our multi-view strategy on diverse public datasets, achieving strong performance in all domains. We demonstrate the effectiveness of representations learned from an unlabeled input collection on downstream tasks with training a linear classifier over pretrained network, usefulness in low-data regime, transfer learning, and cross-validation. Our methodology achieves competitive performance with fullysupervised networks, and it outperforms pre-training with autoencoders in both central and federated contexts. Notably, it improves the generalization in a semi-supervised setting as it reduces the volume of labeled data required through leveraging self-supervised learning.

Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things

“…Since annotating large-scale data can be prohibitively expensive and time-consuming, USL and SSL can be useful for continually improving models in AIoT systems by harvesting the large-scale unlabeled data collected by massive numbers of sensors [196]. Besides, the multi-modal data from heterogeneous sensors (e.g., RGB/infrared/depth camera, IMU, Lidar, microphone) can be used to design cross modal-based pretext tasks (e.g., by leveraging audio-visual correspondence and ego-motion) and free semantic label-based pretext task (e.g., by leveraging depth estimation and semantic segmentation) for self-supervised learning [197].…”

Elastic AI: system support for adaptive machine learning in pervasive computing systems

“…Federated learning allows to adapt an AI by retraining it at runtime (Ek et al 2020(Ek et al , 2021Saeed et al 2020;Konečnỳ et al 2016). Existing approaches do not tackle the aspects of AI software component deployment.…”

An Advanced Stress Detection Approach based on Processing Data from Wearable Wrist Devices

“…In the second part, a transfer learning approach from [13] was used for the supervised classification of affective states by using previously created features. Another study that is based on applying SSL techniques to the WESAD data set is presented in [14]. The paper uses a "pretext task" to train the model without using labeled data, where it must be determined whether the raw data and the wavelet transformations are temporally aligned.…”

“…The paper uses a "pretext task" to train the model without using labeled data, where it must be determined whether the raw data and the wavelet transformations are temporally aligned. The proposed model in [14] is evaluated in two ways: using a linear classifier on top of the SSL component and assessing the number of used samples for the supervised learning process. The first evaluation approach includes a direct comparison of the features created by SSL with the features designed with expert knowledge, as in [6].…”

“…As in [9], this research will also seek to avoid intrusive sensors for collecting data and will focus only on wrist sensors available on commercial smartwatches. Additionally, good practices in work with SSL models in treating the WESAD data from [12,14] will be used as a starting point for developing a novel SSL methodology. A review of exploited methods for collecting, processing, and evaluating data collected by wearables (smartwatches and bands) is presented in [16].…”