The COVID-19 pandemic has presented many challenges to caregivers of children. Families with children with attention-deficit/hyperactivity disorder (ADHD) and/or autism spectrum disorder (ASD) are an understudied but potentially vulnerable population to changes during the outbreak. As such, the aim of this study was to contrast quality of life for caregivers of children with ADHD and/or ASD, before and during the pandemic, compared to caregivers of neurotypical (NT) children. Total, Parent Health-Related Quality of Life, and Family Functioning Summary Scores from the Family Impact Module of the Pediatric Quality of Life InventoryTM were contrasted among caregivers of children with ADHD, ASD, comorbid ADHD and ASD, and NT development. For all scores, caregivers of ADHD and/or ASD children reported lower quality of life, both before and during the pandemic, in comparison to caregivers of NT children. For all diagnoses, quality of life decreased during the pandemic, but caregivers of children with ADHD and/or ASD reported a greater decrease in quality of life than caregivers for NT children. There are limitations to this study in terms of the composition of the sample and the survey methodology, but we are able to conclude that caregivers of children with ADHD and/or ASD have been disproportionately affected by the pandemic, and it is imperative that these families receive additional resources and support to improve their quality of life.