Under the environment of mass-customization, valid samples for a specific target object become fewer and fewer. Thus, Transfer Learning (TL) is widely applied to those data-consuming Deep Learning (DL) and Machine Learning (ML) models to avoid recollecting high-volume training samples from scratch. However, TL-based methods used in fault diagnosis for mechanical rotating components are still in the infancy phase. Poor practicality and usability problems increase the total preparing time. Without online diagnosis, there might be more unknown risks during the preparing time of a TL method. To reduce the preparing time of existing TL techniques, this paper proposes a TL-based Gradual Improvement Scheme (TLGIS) to achieve a systematic online diagnosis of rotating components. TLGIS solves the problem through seamless interactions among five modules of Data Preprocessing (DP), Data Estimating (DE), Data Modeling (DM), Model Improving (MI), and Accuracy Checking, (AC). Via illustrative examples, TLGIS is demonstrated to be reliable and robust using deep Convolutional Neural Networks (CNN) by two public bearing datasets from the Internet and two types of rotating wheel datasets from a practical CNC machine. In total, TLGIS only requires at least 80 samples and at most 250 samples divided into small-size batches to gradually and continuously fine-tune the model with an accuracy of 90% and 99%, respectively, instead of inputting a large-scale dataset all at once, which is unrealistic in a practical production scenario. Compared to conventional approaches, TLGIS efficiently saves at most 90% of preparing time.