Video data model is the most important issue in designing a video database system. This paper proposes a hierarchical semantic associative model. It describes video information in three levels: Hierarchical concept tree, scene network and semantic object net. The concept tree can be constructed under different applications, it assists to determine what objects are to be extracted. Time information of video is represented by the scene network which provides rough description of video information. The contents of video shots are represented by semantic objects and their relationships. Through semantic relations among objects which are in different shots, this model can express the semantic correlations between shots. We can retrieve video by time information and other semantic information. This paper also develops algorithms to process different types of video queries: temporal query, similar query, query by prototype.