The COVID-19 pandemic is associated with significant morbidity, mortality, and restrictions on everyday life worldwide. This may be especially challenging for brain tumor patients given increased vulnerability due to their pre-existing condition. Here, we aimed to investigate the quality of life (QoL) in brain tumor patients and relatives in this setting. Over twelve weeks during the first wave of the pandemic (04–07/2020), brain tumor patients and their families from two large German tertiary care centers were asked to complete weekly questionnaires for anxiety, depression, distress, and well-being. Information regarding social support and living conditions was also collected. One hundred participants (63 patients, 37 relatives) completed 729 questionnaires over the course of the study. Compared to relatives, patients showed more depressive symptoms (p < 0.001) and reduced well-being (p = 0.013). While acceptance of lockdown measures decreased over time, QoL remained stable. QoL measures between patients and their families were weakly or moderately correlated. The number of social contacts was strongly associated with QoL. Age, living conditions, ongoing therapy, employment, and physical activity were other predictors. QoL is correlated between patients and their families and heavily depends on social support factors, indicating the need to focus on the entire family and their social situation for QoL interventions during the pandemic.