In this paper, the formation control of networks of multiple agents is studied via controllability, where the network is under leader-follower structure with some agents taking the leader role and others being followers interconnected via neighbor-based rule. It is shown that the controllability of a multi-agent system is uniquely determined by the topology structure of interconnection graph, and the investigation of which comes down to that for a multi-agent system with the interconnection graph being connected. Based on these observations, two kinds of interconnection graph topologies are characterized, under which the network of multiple agents is uncontrollable, revealing to some extent how the controllability, and accordingly the formation control, are affected by the interconnection topology between agents. Finally, a necessary and sufficient condition in terms of eigenvector is presented. The results also touch upon the selection of leaders and are illustrated by several examples.