This paper considers structural controllability of multi‐agent systems (MASs) with switching topologies, in which each interaction topology is modeled by a weighted digraph. Under the adopted relative protocol, the MASs with directed switching topologies are modeled by a switched system with parameter interdependence. The technique of linear parameterization, the graph theory, and so forth are applied to tackle the structural controllability of the switched system. A graph‐theoretic interpretation for structural controllability of the system is proposed. Examples with single leader and multiple leaders are presented to illustrate our work.