Affective well-being indicates the changes in the predominance of positive or negative affects of people in response to their daily experiences. For measuring affective well-being, people are usually asked to indicate their affective states in several episodes that they experienced in a day. However, such conventional methods have been problematic in terms of their burdensomeness on participants and the validity of the rating. To overcome these problems, we attempted to introduce a new approach for measuring affective states, based on the combination of the day reconstruction method and the measurement of physiological stress levels by a wristband-type wearable device. As the indicator of physiological stress levels, we used heart rate variability calculated from the data recorded by the device. We examined the interpretability of the physiological stress level as a substitute for affective states by applying this combinational approach to an aging and depopulating village in Japan, because the well-being of the residents in such areas is a matter of public concern. As a result, we could depict the sources of affective well-being and the physiological stressors in the village. We also found a reasonable, but weak, correlation between the scores of affective states and the indicator of physiological stress levels. We discussed the challenges that should be overcome for utilizing the physiological stress level as a substitute for affective state.