Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models

Nityasya, Made Nindyatama; Wibowo, Haryo Akbarianto; Chevi, Rendi; Prasojo, Radityo Eko; Aji, Alham Fikri

doi:10.48550/arxiv.2201.00558

Cited by 1 publication

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We take a case in the Indonesian research environment as Indonesian is an underrepresented language [2], [3]. The lack of datasets, NLP researchers, and the high computational costs [4], [5] lead to slow NLP research progress in this area.…”

Section: Introductionmentioning

confidence: 99%

“…Lastly, we investigate simple text-shortening methods which produce better performance. There are several variations to compare: (1) original text, (2) removing stopwords, (3) removing punctuation, (4) removing stopwords and punctuation, (5) removing stopwords and low-frequency words, (6) combining head and tail, and ( 7) unique words only. The result indicates that removing stopwords outperforms the other methods.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Simple Hack for Transformers Against Heavy Long-Text Classification on a Time- and Memory-Limited GPU Service

Mutasodirin,

Prasojo,

Abka

et al. 2023

2023 10th International Conference on Advanced Informatics: Concept, Theory and Application (ICAICTA)

View full text Add to dashboard Cite

Many NLP researchers rely on free computational services, such as Google Colab, to fine-tune their Transformer models, causing a limitation for hyperparameter optimization (HPO) in long-text classification due to the method having quadratic complexity and needing a bigger resource. In Indonesian, only a few works were found on long-text classification using Transformers. Most only use a small amount of data and do not report any HPO. In this study, using 18k news articles, we investigate which pretrained models are recommended to use based on the output length of the tokenizer. We then compare some hacks to shorten and enrich the sequences, which are the removals of stopwords, punctuation, low-frequency words, and recurring words. To get a fair comparison, we propose and run an efficient and dynamic HPO procedure that can be done gradually on a limited resource and does not require a longrunning optimization library. Using the best hack found, we then compare 512, 256, and 128 tokens length. We find that removing stopwords while keeping punctuation and low-frequency words is the best hack. Some of our setups manage to outperform taking 512 first tokens using a smaller 128 or 256 first tokens which manage to represent the same information while requiring less computational resources. The findings could help developers to efficiently pursue optimal performance of the models using limited resources.

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%