In the field of cultural heritage, a significant amount of research has been conducted to allow domain experts to represent semantic data while keeping spatial references. The organization of architectural shapes in significant concepts, in the context of semantic annotation, is carried out in various ways. This chapter analyzes them, highlighting some substantial questions: the semantic ambiguity and the uncertainty. These questions are addressed after a careful reflection on the theoretical basis for a semantic reading of architecture. The chapter proposes some operative solutions: the use of an international thesaurus to resolve linguistic ambiguity; the coding of semantic maps to solve the annotation uncertainty.