This study compared the association between working hours and self-rated health (SRH) according to sex, socioeconomic status, and working conditions. In all, 25,144 participants were selected from the Korea National Health and Nutrition Examination Survey (KNHANES), conducted from 2010 to 2018. The risks of poor SRH, according to working hours, were investigated using multiple logistic regression. Both short and long working hours were associated with poor SRH. Men working short hours and women working long hours were at risk of poor SRH. Workers with fewer than nine years of education were at risk of poor SRH when working short hours, whereas workers with more than nine years of education were at risk when working long hours. Similarly, simple laborers were at risk of poor SRH when working short hours, while managers and professional workers were at risk when working long hours. When working for short hours, paid employees were at risk of poor SRH. Workers with a non-fixed work schedule showed no risk of poor SRH when working long or short hours. In conclusion, workers working short hours with low education and workers working long hours with high education were at risk of poor SRH. Working conditions were significantly related to the association between SRH and working hours.