“…Self-training has been studied for many years (Yarowsky, 1995;Riloff and Wiebe, 2003;Rosenberg et al, 2005) and widely adopted in many NLP tasks including speech recognition (Kahn et al, 2020;Park et al, 2020), parsing (McClosky et al, 2006McClosky and Charniak, 2008), and pre-training (Du et al, 2021). Self-Training suffers from inaccurate pseudo labels (Arazo et al, 2020(Arazo et al, , 2019Hu et al, 2021a) especially when the teacher model is trained on insufficient and unbalanced datasets. To address this problem, (Pham et al, 2020;Wang et al, 2021b;Hu et al, 2021a) propose to utilize the performance of the student model on the held out labeled data as a Meta Learning objective to update the teacher model or improve the pseudo-label generation process.…”