We discuss how to formulate lattice gauge theories in the Tensor Network language. In this way we obtain both a consistent-truncation scheme of the Kogut-Susskind lattice gauge theories and a Tensor Network variational ansatz for gauge invariant states that can be used in actual numerical computations. Our construction is also applied to the simplest realization of the quantum link models/gauge magnets, and provides a clear way to understand their microscopic relation with the Kogut-Susskind lattice gauge theories. We also introduce a new set of gauge invariant operators that modify continuously Rokshar-Kivelson wave functions, and can be used to extend the phase diagrams of known models. As an example we characterize the transition between the deconfined phase of the Z2 lattice gauge theory and the Rokshar-Kivelson point of the U(1) gauge magnet in 2D in terms of entanglement entropy. The topological entropy serves as an order parameter for the transition, but not the Schmidt gap.Tensor Network (TN) techniques are starting to play an important role in our understanding of many-body quantum systems, both on the lattice and in the continuum. They can be used as a framework to classify the phases of quantum matter [1-3], or as powerful numerical ansatz in actual computations of 1D [4, 5] and 2D strongly correlated quantum magnets [6-8], fermionic systems [9, 10], or anyonic systems [11,12]. They have also recently made their way into quantum chemistry as computational tool to study the structure of molecules from the first principles [13,14].While numerical simulations based on Monte Carlo (MC) are still the most successful techniques in some of these fields, TNs start to provide viable alternatives to them, particularly in those contexts where MC has troubles, such as the physics of frustrated anti-ferromagnets [15][16][17], and the real time evolution of out of equilibrium systems [18][19][20].At present, the main limitation of numerical TN techniques is that the cost of the simulations increases rapidly with the amount of correlations in the system (which is encoded in the bond dimension D of the elementary tensors), and thus TNs tend to be biased towards weakly correlated phases.However, the steady improvement of the TN algorithms [21,22] makes us confident that these limitations will soon be overcome, and as a consequence TN will become more and more useful in the physics of quantum many-body systems. Among interesting quantum manybody systems, we focus here on gauge theories, a context in which TN have recently made a spectacular debut [23][24][25][26][27].Gauge theories (GT) [28] describe three of the four fundamental interactions (electromagnetic, weak, and strong interactions). In particular, strong interactions, * luca.tagliacozzo@icfo.es † alessio.celi@icfo.es ‡ maciej.lewenstein@icfo.es are described by an SU (3) gauge theory, called Quantum Chromo-Dynamics (QCD) [29]). GT also allow to understand emergent phenomena at low energies in condensed matter systems, e.g. anti-ferromagnets [30] and high-temper...