Online Continual Learning for Human Activity Recognition

Schiemer, Martin; Fang, Lei; Dobson, Simon; Ye, Juan

doi:10.2139/ssrn.4357622

Cited by 1 publication

(1 citation statement)

References 60 publications

(89 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This will directly affect any conclusions extracted from the IP-dynamics because high bias prevents recognizing the existence of fitting or compression phases, whereas high variance leads to inconsistent results across different numerical experiments. Indeed, with different mutual information estimators, researchers drew diverse or opposite conclusions about trends in IP-dynamics [8,[11][12][13][14][15][16][17][18][19][20][21][22][23][24][25]. For instance, Saxe et al [24] argued that the reported phenomena of fitting and compression in Shwartz et al's study [8] are highly dependent on the simple binning MI estimator setup adopted.…”

Section: Introductionmentioning

confidence: 99%

On Neural Networks Fitting, Compression, and Generalization Behavior via Information-Bottleneck-like Approaches

Lyu

Aminian

Rodrigues

2023

Entropy

View full text Add to dashboard Cite

It is well-known that a neural network learning process—along with its connections to fitting, compression, and generalization—is not yet well understood. In this paper, we propose a novel approach to capturing such neural network dynamics using information-bottleneck-type techniques, involving the replacement of mutual information measures (which are notoriously difficult to estimate in high-dimensional spaces) by other more tractable ones, including (1) the minimum mean-squared error associated with the reconstruction of the network input data from some intermediate network representation and (2) the cross-entropy associated with a certain class label given some network representation. We then conducted an empirical study in order to ascertain how different network models, network learning algorithms, and datasets may affect the learning dynamics. Our experiments show that our proposed approach appears to be more reliable in comparison with classical information bottleneck ones in capturing network dynamics during both the training and testing phases. Our experiments also reveal that the fitting and compression phases exist regardless of the choice of activation function. Additionally, our findings suggest that model architectures, training algorithms, and datasets that lead to better generalization tend to exhibit more pronounced fitting and compression phases.

show abstract