Abstract. The automorphism groups Aut(C(G, X ))and Aut (CM(G, X, p)) of a Cayley graph C(G, X ) and a Cayley map CM (G, X, p) both contain an isomorphic copy of the underlying group G acting via left translations. In our paper, we show that both automorphism groups are rotary extensions of the group G by the stabilizer subgroup of the vertex 1 G . We use this description to derive necessary and sufficient conditions to be satisfied by a finite group in order to be the (full) automorphism group of a Cayley graph or map and classify all the finite groups that can be represented as the (full) automorphism group of some Cayley graph or map.