A Hybrid Deep Learning Architecture for Privacy-Preserving Mobile Analytics

Abstract

Internet of Things (IoT) devices and applications are being deployed in our homes and workplaces and in our daily lives. These devices often rely on continuous data collection and machine learning models for analytics and actuations. However, this approach introduces a number of privacy and efficiency challenges, as the service operator can perform arbitrary inferences on the available data. Recently, advances in edge processing have paved the way for more efficient, and private, data processing at the source for simple tasks and lighter models, though they remain a challenge for larger, and more complicated models. In this paper, we present a hybrid approach for breaking down large, complex deep neural networks for cooperative, privacy-preserving analytics. To this end, instead of performing the whole operation on the cloud, we let an IoT device to run the initial layers of the neural network, and then send the output to the cloud to feed the remaining layers and produce the final result. We manipulate the model with Siamese fine-tuning and propose a noise addition mechanism to ensure that the output of the user's device contains no extra information except what is necessary for the main task, preventing any secondary inference on the data. We then evaluate the privacy benefits of this approach based on the information exposed to the cloud service. We also asses the local inference cost of different layers on a modern handset. Our evaluations show that by using Siamese fine-tuning and at a small processing cost, we can greatly reduce the level of unnecessary, potentially sensitive information in the personal data, and thus achieving the desired trade-off between utility, privacy and performance.

Edge Intelligence: Paving the Last Mile of Artificial Intelligence With Edge Computing

“…With this feature, Transfer Learning is employed in many pieces of research and inspires the design of some frameworks. Osia et al [60], which we have mentioned on Sec. IV-C4 , use Transfer Learning to determine the degree of generality and particularity of a private feature.…”

“…In simulating human intelligence, AI systems typically demonstrate at least some of the following behaviors associated with human intelligence: planning, learning, reasoning, problem-solving, knowledge representation, perception, motion, and manipulation and, to a lesser extent, social intelligence and creativity. During the past 60 year's development, AI has experienced rise, fall and again rise and fall. The latest rise of AI after 2010's was partially due to the breakthroughs made by deep learning, a method that has achieved human-level accuracy in some interesting areas.…”

“…• First train a base network (teacher network) on a base dataset and task and then transfer the learned features to a second target network (student network) to be trained on a target dataset and task • The transition from generality to specificity [59], [60], [62] Gossip Training • Random gossip communication among devices • Full asynchronization and total decentralization • Preserve privacy [63]- [66] data originated by multiple clients. Rather than aggregating the raw data to a centralized datacenter for training, federated learning [45] leaves the raw data distributed on the clients (e.g., mobile devices), and trains a shared model on the server by aggregating locally-computed updates.…”

“…Osia et al [60] introduce a hybrid user-cloud framework on the privacy issue, which utilizes a private-feature extractor as its core component and breaking down large, complex deep models for cooperative, privacy-preserving analytics. In this framework, the feature extractor module is properly designed to output the private feature constrained to keeping the primary information while discarding all the other sensitive information.…”

Deep learning on mobile devices: a review

“…This is particularly true if the data are stored and processed in the cloud. A hybrid deep learning architecture is recently proposed for privacypreserving mobile analytics [26].…”