We show that for any natural number N there exists a right-angled Artin group A Γ for which Out(Aut(A Γ )) has order at least N . This is in contrast with the cases where A Γ is free or free abelian: for all n, Dyer-Formanek and Bridson-Vogtmann showed that Out(Aut(F n )) = 1, while Hua-Reiner showed |Out(Aut(Z n ))| ≤ 4. We also prove the analogous theorem for Out(Out(A Γ )). These theorems fit into a wider context of algebraic rigidity results in geometric group theory. We establish our results by giving explicit examples; one useful tool is a new class of graphs called austere graphs.