Lean evolutionary machine learning by multitasking simpler and hard tasks

Abstract

I would also like to thank my Thesis Advisory Committee (TAC) for their valuable insights and suggestions for my research. Many thanks to Ray Lim, Chen Zhe, and other researchers who I have met during my Ph.D journey. They have always kept my morale high and played a significant role in my personal development.Special thanks to Ms. Lim Siew Hoon and Mr. Suvindaren Subramaniam for all the opportunities given to me in my professional career.Finally, words cannot express my gratitude to my partner, May, my cat Bobby and my parents for their unwavering support and understanding throughout my Ph.D

Benthic visual adaptation by fine-tuning light sensitivity in Japanese flounder (Paralichthys olivaceus)

“…Constructing versatile representations Reusing or transferring features across related tasks has been commonplace for more than one decade (Collobert et al, 2011;Bottou, 2011;Sharif Razavian et al, 2014) and plays a fundamental role in the appeal of foundational models (Bommasani et al, 2021a). However, once the optimization process has identified a set of features that is sufficient to achieve nearoptimal performance on the training set, additional features are often discarded because they do not bring an incremental benefit to the training error, despite the fact that they may independently carry useful information (Zhang & Bottou, 2023).…”

“…assumption, favoring solutions with sparse representations has well-known benefits on the generalization performance. Yet, several authors (Zhang et al, 2022;Zhang & Bottou, 2023;Chen et al, 2023) make the point that scenarios involving multiple distributions are best served by "richer representations", even richer than those constructed without sparsity regularization using the usual stochastic gradient learning procedure.…”

“…Although the precise meaning of "richer representation" is often unclear, Zhang & Bottou (2023) point to the body of evidence suggesting an implicit sparsity bias in deep networks training with stochastic gradient descent (Andriushchenko et al, 2023;Blanc et al, 2020). In this setup, a feature only survives when it brings an incremental training error advantage relative to what can be achieved using all the other features already present in the network.…”

“…In this setup, a feature only survives when it brings an incremental training error advantage relative to what can be achieved using all the other features already present in the network. Zhang & Bottou (2023) then argue that a feature may not be incrementally useful under the training distribution but (a) might be incrementally useful when the data follows a different distributions of interest, or (b) might be useful under the training distribution when added to certain subsets of the other existing features instead of all of them.…”

“…In the present work, we adopt this empirical setup. However, instead of enriching the representations by constructing ensembles (Zhang & Bottou, 2023) or averaging weights (Ramé et al, 2022b;a;Wortsman et al, 2022b), we simply fine-tune using very large dropout levels, randomly masking about 90% of the units in the representation layer. We find that this simple approach exceeds the performance of either ensembles or weight-averagings.…”