Understanding AdamW through Proximal Methods and Scale-Freeness

Zhuang, Zhenxun; Liu, Mingrui; Cutkosky, Ashok; Orabona, Francesco

doi:10.48550/arxiv.2202.00089

Cited by 12 publications

(13 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Decay Weight by Proximation. As observed in AdamW [3,57], decoupling the optimization objective and simple-type regularization (e.g. 2 regularizer) can largely improve the generalization performance.…”

Section: Adaptive Nesterov Momentum Algorithmmentioning

confidence: 98%

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

Xie¹,

Zhou²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

Adaptive gradient algorithms [1][2][3][4] borrow the moving average idea of heavy ball acceleration to estimate accurate first-and second-order moments of gradient for accelerating convergence. However, Nesterov acceleration which converges faster than heavy ball acceleration in theory [5] and also in many empirical cases [6] is much less investigated under the adaptive gradient setting. In this work, we propose the ADAptive Nesterov momentum algorithm, Adan for short, to effectively speedup the training of deep neural networks. Adan first reformulates the vanilla Nesterov acceleration to develop a new Nesterov momentum estimation (NME) method, which avoids the extra computation and memory overhead of computing gradient at the extrapolation point. Then Adan adopts NME to estimate the first-and second-order moments of the gradient in adaptive gradient algorithms for convergence acceleration. Besides, we prove that Adan finds an -approximate first-order stationary point within O −3.5 stochastic gradient complexity on the nonconvex stochastic problems (e.g. deep learning problems), matching the bestknown lower bound. Extensive experimental results show that Adan surpasses the corresponding SoTA optimizers on both CNNs and transformers, and sets new SoTAs for many popular networks and frameworks, e.g. ResNet [7], ConvNext [8], ViT [9], Swin [10], MAE [11], LSTM [12], TransformerXL [13] and BERT [14].More surprisingly, Adan can use half of the training cost (epochs) of SoTA optimizers to achieve higher or comparable performance on ViT and ResNet, etc, and also shows great tolerance to a large range of minibatch size, e.g. from 1k to 32k. We hope Adan can contribute to the development of deep learning by reducing training cost and relieving engineering burden of trying different optimizers on various architectures. Code is released at https://github.com/sail-sg/Adan. * Equal contribution. Xingyu did this work during an internship at Sea AI Lab.Preprint. Under review.

show abstract

Section: Adaptive Nesterov Momentum Algorithmmentioning

confidence: 98%

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

Xie¹,

Zhou²,

Li³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, this method heavily relies on the choice of optimization algorithm [13], as there is a risk of getting stuck in local minima during gradient computation, leading to the model's inability to learn and improve prediction/recognition quality (vanishing gradients). To address this, we employed the AdamW [14] optimizer, one of the state-of-the-art methods, which leverages information about the learning rate history to approximate the direction of the anti gradient while incorporating momentum to expedite the convergence of our function. This optimizer significantly improves model training; however, it is sensitive to the choice of the learning rate.…”

Section: Transformer Modelsmentioning

confidence: 99%

Transformer-Based Classification of User Queries for Medical Consultancy

Lyutkin,

Pozdnyakov,

Soloviev

et al. 2024

ARC

View full text Add to dashboard Cite

The need for skilled medical support is growing in the era of digital healthcare. This research presents an innovative strategy, utilizing the RuBERT model, for categorizing user inquiries in the field of medical consultation with a focus on expert specialization. By harnessing the capabilities of transformers, we fine-tuned the pre-trained RuBERT model on a varied dataset, which facilitates precise correspondence between queries and particular medical specialisms. Using a comprehensive dataset, we have demonstrated our approach’s superior performance with an F1-score of over 91.8%, calculated through both cross-validation and the traditional split of test and train datasets. Our approach has shown excellent generalization across medical domains such as cardiology, neurology and dermatology. This methodology provides practical benefits by directing users to appropriate specialists for prompt and targeted medical advice. It also enhances healthcare system efficiency, reduces practitioner burden, and improves patient care quality. In summary, our suggested strategy facilitates the attainment of specific medical knowledge, offering prompt and precise advice within the digital healthcare field.

show abstract

“…This article makes some fine-tuning to the BERT [4] model, adjusting the learning rate to 1 × 10 −5 , batchsize to 12, epochs to 10, and the optimizer uses AdamW [7,8] to run the data in data parallel to accelerate the model.…”

Section: Implementation Detailsmentioning

confidence: 99%

Analyzing film and drama reviews: Distinguishing trolls from genuine audience feedback based on the BERT model

2024

ACE

View full text Add to dashboard Cite

With the expanding influence of the Internet, an increasing number of individuals rely on viewer reviews to make informed decisions about whether to watch a movie or TV series. However, the prevalence of manipulated or "navy" reviews, employed by companies to boost their products' reputation, has created a significant challenge. While numerous studies have dissected film and drama reviews, a notable gap exists in discerning genuine audience feedback from deceptive ones. This article's research focus is on evaluating the model's capacity to effectively differentiate between authentic audience comments and navy reviews and delving into the complexities encountered when the model assesses comments, as well as highlighting the disparities between model-generated judgments and human assessments. This article first collects a large amount of different types of comment data, annotates these data, and then uses these data to train and fine tune the BERT model. Finally, the results are obtained and analyzed to determine the reasons. This article found that the accuracy rate of the model's judgment comments is around 71.08%, which is more accurate and stable. However, there are still some issues when judging comments with emojis and emoticons, and certain data is needed to support the judgment of comments for different movies or dramas. There are also certain issues with the dataset, as the data is manually annotated, and the annotation of the dataset itself may also be influenced by the annotator, which may lead to inaccurate judgments.

show abstract

Understanding AdamW through Proximal Methods and Scale-Freeness

Cited by 12 publications

References 26 publications

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

Transformer-Based Classification of User Queries for Medical Consultancy

Analyzing film and drama reviews: Distinguishing trolls from genuine audience feedback based on the BERT model

Contact Info

Product

Resources

About