Let $N$ be a prime number, and $X$ a curve that is intermediate to the cover $X_1(N)\rightarrow X_0(N)$. We study the automorphism group of $X$, and prove that in most cases it is generated by the Galois group of the cover $X\rightarrow X_0(N)$, and any lift of the Atkin–Lehner involution to $X$.