Faithfully representing chemical environments is essential for describing materials and molecules with machine learning approaches. Here, we present a systematic classification of these representations and then investigate: (i) the sensitivity to perturbations and (ii) the effective dimensionality of a variety of atomic environment representations, and over a range of material datasets. Representations investigated include Atom Centred Symmetry Functions, Chebyshev Polynomial Symmetry Functions, Smooth Overlap of Atomic Positions, Many-body Tensor Representation and Atomic Cluster Expansion. In both areas (i) and (ii), we find significant variation across the different representations in these tests, pointing to shortcomings and the need for further improvements.