SummaryDevelopment of intelligent robots and rapid increase of aged societies have brought serious necessity of such systems that should facilitate mutual translation of sensory data and linguistic expressions. They are expected to help people, especially with some defected sense-organ, by translating sensory data into words such as "Pungent smell is sensed in the refrigerator!", and otherwise enable people to order a robot to work by words such as "Search the room for a varicolored object."For the purpose to develop such a system, the authors have recently analyzed and described the concepts of Japanese words for "color" and "lightness" based on MIDST(Mental image directed semantic theory) proposed by Yokota,M. et al.The analysis and description of the word concepts were performed approximately in the process as follows. Firstly, each word representing a specific color or lightness(" (=red)", " (=dark)", etc) was associated with a set of specific coordinates (point or range) of the color solid and its concept was defined as such a set of coordinates. Secondly, the words concerning temporal change or spatial distribution of color or lightness(" (=redden)", " (=gradate)", etc) were described as spatiotemporal relations among coordinates of the color solid. Thirdly, a computer system working with image input devices was constructed in order to ground words on real sensory data of color and lightness via the coordinates of the color solid in an interactive way with a human instructor, and has been found a fairly good success.
The Mental Image Directed Semantic Theory (MIDST) has proposed an intermediate knowledge representation scheme based on an omnisensual mental image model. This paper presents a formal language for describing multimedia contents, L md , whose syntax and semantics are based on MIDST and its application to linguistic interpretation of human motion data obtained through a motion capture system, which we think will serve for linguistic summarization of immense amount of time-sequenced non-linguistic data such as movies.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.