Stroke survivors majorly suffered from post-stroke depression (PSD). The PSD diagnosis is commonly performed based on the clinical cut-off for psychometric inventories. However, we hypothesized that PSD involves spectrum symptoms (e.g., apathy, depression, anxiety, and stress domains) and severity levels. Therefore, instead of using the clinical cut-off, we suggested a data-driven analysis to interpret patient spectrum conditions. The patients’ psychological conditions were categorized in an unsupervised manner using the k-means clustering method, and the relationships between psychological conditions and quantitative lesion degrees were evaluated. This study involved one hundred sixty-five patient data; all patients were able to understand and perform self-rating psychological conditions (i.e., no aphasia). Four severity levels—low, low-to-moderate, moderate-to-high, and high—were observed for each combination of two psychological domains. Patients with worse conditions showed the significantly greater lesion degree at the right Rolandic operculum (part of Brodmann area 43). The dissimilarities between stress and other domains were also suggested. Patients with high stress were specifically associated with lesions in the left thalamus. Impaired emotion processing and stress-affected functions have been frequently related to those lesion regions. Those lesions were also robust and localized, suggesting the possibility of an objective for predicting psychological conditions from brain lesions.