To m ake accu rate deci sions based on i maging data, radiolog ists must as sociate t he viewed i maging data with t he corresponding anatomical structures. Furthermore, given a disease hypothesis possible image findings which verify the hypothesis must be considered and where and how they are expressed in the viewed images. If rare anatomical variants, rare pathologies, unfamiliar protocols, or am biguous findings are pres ent, external knowledge sources such as medical encyclopedias are con sulted. T hese s ources are acces sed u sing keywords t ypically de scribing a natomical s tructures, image findings, pathologies.In th is paper we pres ent o ur v ision of how a patie nt's i maging data can be a utomatically e nhanced with an atomical knowledge as well as knowledge about image findings. On one hand, we propose the automatic annotation of the images with labels from a standard anatomical ontology. These labels are used as keywords for a medical encyclopedia such as STATdx to acces s an atomical des criptions, inf ormation abou t path ologies an d i mage findings. O n th e ot her h and we envision e ncyclopedias to contain links to region-and finding-specific i mage processing algorithms. T hen a f inding is evaluated on an image by applying the respective algorithm in the associated anatomical region.Towards realization of our vision, we present our method and results of automatic annotation of anatomical structures in 3D MRI brain i mages. Thereby we develop a co mplex surface mesh model incorporating major structures of the brain and a model-based segmentation method. We dem onstrate the validity b y a nalyzing t he results of several training and segmentation experiments with clinical data focusing particularly on the visual pathway.