Background Caregivers play a vital role in caring for the aging population, however the occurrence of violence against the caregiver is an increasing area of concern. This study aimed to investigate the prevalence of violence against the primary caregivers of community dwelling older adults with chronic diseases, and to determine the factors associated with violence and its association with caregiver outcomes. Methods A cross-sectional study was conducted. HITS questionnaire, the 22-item Zarit Burden Interview and Patient Health Questionnaire-9 were used to assess violence against caregiver, caregiver burden and depression, respectively. Results Out of 123 caregivers of older adults, the overall prevalence of violence was 28.46%. Independent variables which could be the protective factors for violence against caregiver included higher ADL, older age of caregiver, and being a relative. The patient characteristic that is a potential risk factor for violence against caregiver was having cancer as a principal diagnosis. Statistically significant associations were found between violence and caregiver burden (aOR 4.94, p 0.004) and depression (aOR 7.03, p 0.006). Conclusion Violence against caregivers of older adults is not uncommon. Experiencing violence was found to be associated with caregiver outcomes including depression and caregiver burden. Therefore, this important issue must not be ignored.