The possible ground states of the undoped and doped Kitaev-Heisenberg model on a triangular lattice are studied. For the undoped system, a combination of the numerical exact diagonalization calculation and the four-sublattice transformation analysis suggests one possible exotic phase and four magnetically ordered phases, including a collinear stripe pattern and a noncollinear spiral pattern in the global phase diagram. The exotic phase near the antiferromagnetic (AF) Kitaev point is further investigated using the Schwinger-fermion mean-field method, and we obtain an energetically favorable Z 2 chiral spin liquid with a Chern number ±2 as a promising candidate. At finite doping, we find that the AF Heisenberg coupling supports an s-wave or a + − d d i x y xy 2 2 -wave superconductivity (SC), while the AF and the ferromagnetic Kitaev interactions favor a + − d d i x y xy 2 2 -wave SC and a time-reversal invariant topological p-wave SC, respectively. Possible experimental realizations and related candidate materials are also discussed.
IntroductionRecently, there has been enormous interest in the physics of the spin-1/2 Kitaev model on a honeycomb lattice [1], which has an exact Z 2 spin-liquid (SL) ground state (GS) supporting fractionalized excitations. One possible route to realize this highly anisotropic spin model is to include a strong relativistic spin-orbit coupling (SOC) in Mott insulators [2,3]. Indeed, the interplay of SOC and electron interactions [4][5][6][7][8] gives rise to many novel phases [9][10][11][12][13], especially for the so-called relativistic Mott insulators (RMIs) whose physics may drastically differ from that of Mott insulators with weak SOC (e.g., cuprates) [3,4,14,15]. Of particular interest are d 5 transition metal oxides, such as iridates A 2 IrO 3 (A = Na, Li) [16][17][18], where Na 2 IrO 3 is interpreted as a novel RMI [19] and may also host the quantum spin Hall effect [20,21]. The Kitaev-Heisenberg (KH) model on a honeycomb lattice, which has a rich phase diagram containing unconventional magnetic and Kitaev SL phases [22,23], has been proposed to capture the low-energy properties of A 2 IrO 3 [24,25]. Meanwhile, experiments confirm a longrange zigzag spin order in Na 2 IrO 3 [26][27][28][29], which is a natural GS of the KH model [22]. In addition, there are also studies on the d 4 compound Li 2 RhO 3 , suggesting Li 2 RhO 3 as a possible RMI with a spin-glass GS [30]. Theoretical studies also show that carrier doping into RMIs can induce unconventional superconducting pairings as well as the topological superconductivity (SC) [31][32][33][34][35][36].In fact, the KH model can be generalized to the triangular lattice [37,38]. Similar to the microscopic origin of the honeycomb KH model for A 2 IrO 3 , the triangular KH model can emerge from a class of ABO 2 -type (where A and B are alkali and transition metal ions, respectively) layered compounds [2,3], due to the joint effect of strong SOC, Coulomb interaction, orbital degeneracy, t g 2 5 configuration, and 90°-bonding geometry [...