One-third of infertile couples may have a male factor present. Illicit drug use can be an important cause of male factor infertility and includes use of anabolic-androgenic steroids, marijuana, opioid narcotics, cocaine, and methamphetamines. The use of these illicit drugs is common in the United States, with a yearly prevalence rate for any drug consistently higher in males compared with females. We aim to provide a review of recent literature on the prevalence and effects of illicit drug use on male fertility and to aid health professionals when counseling infertile men whose social history suggests illicit drug use. Anabolic-androgenic steroids, marijuana, cocaine, methamphetamines, and opioid narcotics all negatively impact male fertility, and adverse effects have been reported on the hypothalamic-pituitary-testicular axis, sperm function, and testicular structure. The use of illicit drugs is prevalent in our society and likely adversely impacting the fertility of men who abuse drugs.