In tolerancing activities focusing on the allocation of geometrical tolerances, many critical issues originate from the non-optimal assignment of responsibilities among the organization units involved. This paper aims to depict relations between different tolerancing activities and relevant specifications, assigning them to the proper actor and, therefore, expanding the ISO 8015:2011 “responsibility principle”. A classification among tolerancing activities, specifications, and media is proposed; a horizontal hierarchical framework among functional, manufacturing, and verification specifications and a vertical hierarchical framework along the supply chain are discussed. Examples of both hierarchical structures are presented.