Investigation on a hierarchical structure of fabric appearance-related perceptual and affective experience afford further understanding how to approach the establishment of assessment on fabric appearance. In this study, we tried to representing the assessment process based on three layers (basic appearance, middle appearance and upper appearance). And the assessment process was defined based on causal relationship among the 27 adjectives resulting from sensory assessment with 6 black fabrics by experts panels. Conducting factor analysis enabled the middle appearance to define with adjectives included in basic appearance. As a result, it was found that the middle appearance was constituted with 4 factors, (i.e., 'macroscopic or microscopic brightness', 'softness', and 'roughness'). Furthermore, establishing multiple regression equation enabled the upper appearance to define by factors involved with the middle appearance. It was found that the upper appearance was impacted by magnitude of 'macroscopic brightness' negatively, and impacted by magnitude of 'softness' and 'microscopic brightness' positively.