Abstract. Medical image annotation is playing an increasingly important role in clinical diagnosis and medical research. Existing medical image annotation is faced with many demands and challenges. 1) The emergence and sharp increasing speed of multi-dimensional medical images. 2) Image annotation includes not only text annotation, but also graphical annotation, clinical diagnostic information and image content features information. 3) Uneven distribution of medical resources, which makes difficult to aggregate group intelligence from a much larger scale of distributed experts. Most of the present study is texted based within hospitals on single images annotation. It is difficult to organize and manage unstructured medical image annotation and collaborative sharing information. This paper dedicated to the research on collaborative web-based multi-dimensional medical image annotation and retrieval in order to address these problems, overcome the shortcoming of traditional thin client and facilitate medical experts in different locations to exchange views and comments,. It proposed 1) a system architecture that provides authoring, storing, querying, and exchanging of annotations, and supports webbased collaboration. 2) 2D multi-frame and 3D medical image collaborative annotation data model. 3) Collaborative annotation mechanisms.