Infertility has been reported as one of the most common reproductive impairments, affecting nearly one in six couples worldwide. A large proportion of infertility cases are diagnosed as idiopathic, signifying a deficit in information surrounding the pathology of infertility and necessity of medical intervention such as assisted reproductive therapy. Small noncoding RNAs (sncRNAs) are well-established regulators of mammalian reproduction. Advanced technologies have revealed the dynamic expression and diverse functions of sncRNAs during mammalian germ cell development. Mounting evidence indicates sncRNAs in sperm, especially microRNAs (miRNAs) and transfer RNA (tRNA)-derived small RNAs (tsRNAs), are sensitive to environmental changes and mediate the inheritance of paternally acquired metabolic and mental traits. Here, we review the critical roles of sncRNAs in mammalian germ cell development. Furthermore, we highlight the functions of sperm-borne sncRNAs in epigenetic inheritance. We also discuss evidence supporting sncRNAs as promising biomarkers for fertility and embryo quality in addition to the present limitations of using sncRNAs for infertility diagnosis and treatment.