Rehabilitation for hospitalized older people can improve their independence for performing activities of daily living (ADL), but determining its appropriateness can be challenging because of inherent limitations in their ADL and short life expectancy. Thus, we aimed to clarify the benefit of rehabilitation among older Japanese patients. We retrospectively evaluated consecutive older patients (age > 65 years) admitted to the rehabilitation unit of a rural community hospital between 1 April 2016 and 31 March 2020. The primary outcome measure was readmission for acute conditions. Of the 732 patients evaluated, 311 patients (42.5%) were readmitted. Readmission was significantly associated with body mass index (BMI) (p < 0.001), dependent condition (p < 0.001), higher cognitive domain scores in the functional independence measure (FIM) (p = 0.019), and polypharmacy (p = 0.026). The most frequent cause of readmission was pyelonephritis (11.9%), followed by pneumonia (10.9%), compression fracture (10.6%), heat stroke (8.4%), and cerebral stroke (8.0%). In conclusion, older Japanese patients discharged from rehabilitation units have lower readmission rates than those previously reported. Thus, better nutritional control, a multidisciplinary approach to the management of cognitive dysfunction, and a decrease in polypharmacy could be associated with improved outcomes among discharged older patients.