Towards Lower Precision Adaptive Filters: Facts From Backward Error Analysis of RLS

Lee, Junkyu; Vandierendonck, Hans

doi:10.1109/tsp.2021.3086355

Cited by 6 publications

(4 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The accuracy of TinyOFL is often limited due to its restricted model capacity. However, the accuracy is not affected by weight quantization if the quantization noise is relatively lower than the data noise [21]. The feature map using quantized weights, fi (x i ), on the input feature x i at the i-th layer can be deviated by δy i due to the weight quantization as follows:…”

Section: B Weight Quantization Effects On Accuracymentioning

confidence: 99%

“…where f i (x i ) is the feature map operation on x i using nonquantized weights. The fi (x i ) generates the quantity δq i based on the backward error analysis [21], [22]:…”

Section: B Weight Quantization Effects On Accuracymentioning

confidence: 99%

“…). We refer to the backward error as the quantization noise since the backward error behaves as the noise on the feature x i [21].…”

Section: B Weight Quantization Effects On Accuracymentioning

confidence: 99%

“…Next, we explore the effects of quantization on the accuracy mathematically by linking the quantization error with data noise [21]. By the backward error analysis [21], [22], the input neurons multiplied by the quantized weights can be expressed as the perturbed input neurons multiplied by the original weights as follows:…”

Section: B Weight Quantization Effects On Accuracymentioning

confidence: 99%

See 3 more Smart Citations

The Effects of Weight Quantization on Online Federated Learning for the IoT: A Case Study

Giménez,

Lee,

Freitag

et al. 2024

IEEE Access

Self Cite

View full text Add to dashboard Cite

Many weight quantization approaches were explored to save the communication bandwidth between the clients and the server in federated learning using high-end computing machines. However, there is a lack of weight quantization research for online federated learning using TinyML devices which are restricted by the mini-batch size, the neural network size, and the communication method due to their severe hardware resource constraints and power budgets. We name Tiny Online Federated Learning (TinyOFL) for online federated learning using TinyML devices in the Internet of Things (IoT). This paper performs a comprehensive analysis of the effects of weight quantization in TinyOFL in terms of accuracy, stability, overfitting, communication efficiency, energy consumption, and delivery time, and extracts practical guidelines on how to apply the weight quantization to TinyOFL. Our analysis is supported by a TinyOFL case study with three Arduino Portenta H7 boards running federated learning clients for a keyword spotting task. Our findings include that in TinyOFL, a more aggressive weight quantization can be allowed than in online learning without FL, without affecting the accuracy thanks to TinyOFL's quasi-batch training property. For example, using 7-bit weights achieved the equivalent accuracy to 32-bit floating point weights, while saving communication bandwidth by 4.6×. Overfitting by increasing network width rarely occurs in TinyOFL, but may occur if strong weight quantization is applied. The experiments also showed that there is a design space for TinyOFL applications by compensating for the accuracy loss due to weight quantization with an increase of the neural network size.

show abstract

Section: B Weight Quantization Effects On Accuracymentioning

confidence: 99%

“…where f i (x i ) is the feature map operation on x i using nonquantized weights. The fi (x i ) generates the quantity δq i based on the backward error analysis [21], [22]:…”

Section: B Weight Quantization Effects On Accuracymentioning

confidence: 99%

“…). We refer to the backward error as the quantization noise since the backward error behaves as the noise on the feature x i [21].…”

Section: B Weight Quantization Effects On Accuracymentioning

confidence: 99%