We propose a variant of the antiferromagnetic XY model on the triangular lattice to study the interplay between the chiral and nematic orders in addition to the magnetic order. The model has a significant bi-quadratic interaction of the planar spins. When the bi-quadratic exchange energy dominates, a large temperature window is shown to exist over which the nematic and the chiral orders co-exist without the magnetic order, thus defining a chiral-nematic state. The phase diagram of the model and some of its critical properties are derived by means of the Monte Carlo simulation.
PACS numbers:Nontrivial orders in frustrated magnets [1] are among the central issues in the field of condensed-matter physics. Besides the conventional magnetic order parameter of spin S i at a site i, there could appear various nontrivial orders such as vector [2,3] and scalar [4, 5] chiral orders [6], and nematic order [7], which might lead to additional phase transitions distinct from the one driven by magnetic order. Even the ground state itself may be characterized solely by these nontrivial orders. This issue is now attracting revived interest from the viewpoint of nontrivial glass transition of spins [8] and multiferroic behaviors [9,10], where the ferroelectricity is induced by the formation of vector spin chirality [11]. One important aspect of this problem is the interplay between the various orders. Usually the nontrivial orders become long ranged when the magnetic order sets in. For example, the spiral spin order naturally implies the vector spin chiral order through S i × S j ∼ S i × S j on the neighboring sites. Therefore, the interesting issue is whether the nontrivial order can become long ranged in the absence of the magnetic order. This issue has been studied theoretically [9], and experimentally in the quasi-one dimensional frustrated magnet [12] where the chiral order appears above the magnetic phase transition. The next important question, we argue, is the interplay between the two nontrivial orders, e.g., chiral and nematic orders, which has not been fully addressed so far.To address this issue, we study a generalized classical XY spin model on a triangular lattice,where θ ij is the angle difference θ i − θ j between the nearest neighbors ij . This model contains the usual frustration in the exchange interaction due to the triangular lattice geometry, together with the possible nematic order induced by the J 2 term. The J 2 = 0 limit has been extensively studied, and it is believed to have two phase transitions at closely spaced critical temperatures[2, 13-15]. The Kosterlitz-Thouless (KT) transition temperature T KT signaling the loss of (algebraic) magnetic order and the melting temperature of the staggered chirality, T χ , are extremely close, (T χ − T KT )/T χ 0.02 at J 2 = 0, hampering the interpretation of the intermediate, T KT < T < T χ phase as the chiral phase in which the chirality is ordered but the magnetism remains disordered. Extension of the XY model to include large J 2 interaction was considered ear...