We explore the phase diagram of the Kitaev-Heisenberg model with nearest neighbor interactions on the honeycomb lattice using the exact diagonalization of finite systems combined with the cluster mean field approximation, and supplemented by the insights from the linear spin-wave and second-order perturbation theories. This study confirms that by varying the balance between the Heisenberg and Kitaev term, frustrated exchange interactions stabilize in this model four phases with magnetic long range order: Néel phase, ferromagnetic phase, and two other phases with coexisting antiferromagnetic and ferromagnetic bonds, zigzag and stripy phases. They are separated by two disordered quantum spin-liquid phases, and the one with ferromagnetic Kitaev interactions has a substantially broader range of stability as the neighboring competing ordered phases, ferromagnetic and stripy, have very weak quantum fluctuations. Focusing on the quantum spin-liquid phases, we study spatial spin correlations and dynamic spin structure factor of the model by the exact diagonalization technique, and discuss the evolution of gapped low-energy spin response across the quantum phase transitions between the disordered spin liquid and magnetic phases with long range order.

Compounds of transition metal ions with strong spin-orbit coupling recently attracted attention due to the possibility to host frustrated bond-dependent anisotropic magnetic interactions. In general, such interactions lead to complex phase diagrams that may include exotic phases, e.g. the Kitaev spin liquid. Here we report on our comprehensive analysis of the global phase diagram of the extended Kitaev-Heisenberg model relevant to honeycomb lattice compounds Na 2 IrO 3 and α-RuCl 3 . We have utilized recently developed method based on spin coherent states that enabled us to resolve arbitrary spin patterns in the cluster ground states obtained by exact diagonalization. Global trends in the phase diagram are understood in combination with the analytical mappings of the Hamiltonian that uncover peculiar links to known models -Heisenberg, Ising, Kitaev, or compass models on the honeycomb lattice -or reveal entire manifolds of exact fluctuation-free ground states. Finally, our study can serve as a methodological example that can be applied to other spin models with complex bond-dependent non-Heisenberg interactions.