An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry

Jiang, Wenxin; Synovic, Nicholas; Hyatt, Matt; Schorlemmer, Taylor R.; Läufer, Konstantin; Lü, Yanbin; Thiruvathukal, George K.; Davis, J. C.

doi:10.1109/icse48619.2023.00206

Cited by 17 publications

(1 citation statement)

References 50 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Increasing the size of the training dataset has limited benefits to the performance of trained models. Jiang et al [46] conducted the first study of pre-trained model reuse. Concretely, they interviewed practitioners from Hugging Face and identified the challenges of pre-trained model reuse, e.g., missing attributes, discrepancies, and model risks.…”

Section: Empirical Study On Ml4codementioning

confidence: 99%

Active Code Learning: Benchmarking Sample-Efficient Training of Code Models

Hu,

Guo,

Xie

et al. 2024

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

The costly human effort required to prepare the training data of machine learning (ML) models hinders their practical development and usage in software engineering (ML4Code), especially for those with limited budgets. Therefore, efficiently training models of code with less human effort has become an emergent problem. Active learning is such a technique to address this issue that allows developers to train a model with reduced data while producing models with desired performance, which has been well studied in computer vision and natural language processing domains. Unfortunately, there is no such work that explores the effectiveness of active learning for code models. In this paper, we bridge this gap by building the first benchmark to study this critical problem -active code learning. Specifically, we collect 11 acquisition functions (which are used for data selection in active learning) from existing works and adapt them for code-related tasks. Then, we conduct an empirical study to check whether these acquisition functions maintain performance for code data. The results demonstrate that feature selection highly affects active learning and using output vectors to select data is the best choice. For the code summarization task, active code learning is ineffective which produces models with over a 29.64% gap compared to the expected performance. Furthermore, we explore future directions of active code learning with an exploratory study. We propose to replace distance calculation methods with evaluation metrics and find a correlation between these evaluation-based distance methods and the performance of code models.

show abstract

Section: Empirical Study On Ml4codementioning

confidence: 99%