Semi-supervised on-device neural network adaptation for remote and portable laser-induced breakdown spectroscopy

Bhardwaj, Kshitij; Gokhale, Maya

doi:10.48550/arxiv.2104.03439

Cited by 2 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Time series data is evident in systems such as IoT devices [13], engines [41], and spacecraft [2,54], where new insights can be gleaned from the large amounts of unmonitored information. Moreover, such systems often suffer from resource constraints, making regular deep learning models unrealistic -for instance, in the Mars rover missions where battery-powered devices are searching for life [5]. Other systems such as satellites contain thousands of telemetry channels that require granular monitoring.…”

Section: Introductionmentioning

confidence: 99%

Sparse Binary Transformers for Multivariate Time Series Modeling

Gorbett

Shirazi

Ray

2023

Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

Compressed Neural Networks have the potential to enable deep learning across new applications and smaller computational environments. However, understanding the range of learning tasks in which such models can succeed is not well studied. In this work, we apply sparse and binary-weighted Transformers to multivariate time series problems, showing that the lightweight models achieve accuracy comparable to that of dense floating-point Transformers of the same structure. Our model achieves favorable results across three time series learning tasks: classification, anomaly detection, and single-step forecasting. Additionally, to reduce the computational complexity of the attention mechanism, we apply two modifications, which show little to no decline in model performance: 1) in the classification task, we apply a fixed mask to the query, key, and value activations, and 2) for forecasting and anomaly detection, which rely on predicting outputs at a single point in time, we propose an attention mask to allow computation only at the current time step. Together, each compression technique and attention modification substantially reduces the number of non-zero operations necessary in the Transformer. We measure the computational savings of our approach over a range of metrics including parameter count, bit size, and floating point operation (FLOPs) count, showing up to a 53× reduction in storage size and up to 10.5× reduction in FLOPs. CCS CONCEPTS• Computing methodologies → Neural networks; Supervised learning.

show abstract

Section: Introductionmentioning

confidence: 99%

Sparse Binary Transformers for Multivariate Time Series Modeling

Gorbett

Shirazi

Ray

2023

Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining

View full text Add to dashboard Cite

show abstract

“…For example, consider the real-world use case of the "Mars rover mission" that uses laser-induced breakdown spectroscopy (LIBS) to search for indicators of microbial life. It is well accepted that endowing the Perseverance rover with DNNs to analyze high-dimensional and complex LIBS spectra offers a huge potential to make scientific breakthroughs [1]. Yet, such an effort is non-existent because: 1) as these devices are battery operated, the model has to be lightweight so it consumes less memory with reduced power consumption, and 2) the model must be able to efficiently handle domain shifts in spectral data caused by environmental or sensor noise.…”

Section: Introductionmentioning

confidence: 99%

A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness

Diffenderfer¹,

Bartoldson²,

Chaganti³

et al. 2021

Preprint

View full text Add to dashboard Cite

Two crucial requirements for a successful adoption of deep learning (DL) in the wild are: (1) robustness to distributional shifts, and (2) model compactness for achieving efficiency. Unfortunately, efforts towards simultaneously achieving Out-of-Distribution (OoD) robustness and extreme model compactness without sacrificing accuracy have mostly been unsuccessful. This raises an important question: "Is the inability to create Compact, Accurate, and Robust Deep neural networks (CARDs) fundamental?" To answer this question, we perform a largescale analysis for a range of popular model compression techniques which uncovers several intriguing patterns. Notably, in contrast to traditional pruning approaches (e.g., fine tuning and gradual magnitude pruning), we find that "lottery ticket-style" pruning approaches can surprisingly be used to create high performing CARDs. Specifically, we are able to create extremely compact CARDs that are dramatically more robust than their significantly larger and full-precision counterparts while matching (or beating) their test accuracy, simply by pruning and/or quantizing. To better understand these differences, we perform sensitivity analysis in the Fourier domain for CARDs trained using different data augmentation methods. Motivated by our analysis, we develop a simple domain-adaptive test-time ensembling approach (CARD-Deck) that uses a gating module to dynamically select an appropriate CARD from the CARD-Deck for each test sample based on their spectral-similarity with test samples. By leveraging complementary frequency biases of different compressed models, the proposed approach builds a "winning hand" of CARDs that establishes a new state-of-the-art on CIFAR-10-C accuracies (i.e., 96.8% clean and 92.75% robust) with dramatically better memory usage than existing non-compressed baselines. We also present theoretical evidences supporting our empirical findings. * equal contribution Preprint. Under review.

show abstract

Semi-supervised on-device neural network adaptation for remote and portable laser-induced breakdown spectroscopy

Cited by 2 publications

References 9 publications

Sparse Binary Transformers for Multivariate Time Series Modeling

Sparse Binary Transformers for Multivariate Time Series Modeling

A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness

Contact Info

Product

Resources

About