Stacking sensitive topological phases in a bilayer Kane-Mele-Hubbard model at quarter filling

Abstract

Layered quasi-two-dimensional systems have garnered huge interest both in the advancement of technology and in understanding emergent physics such as unconventional superconductivity and topological phases. In particular, the study of topological properties in some bilayer systems like transition metal chalcogenides and iridates has been the point of attraction due to comparatively strong spin-orbit coupling of transition metal ions. In this paper, we analyze the topological phases induced by the interplay of electron correlation and spin-orbit coupling in different stacking orders of bilayer honeycomb lattice at quarter filling. Considering the two most common stacking orders, AA and Bernal (AB) stacking, we show that the stacking order plays a crucial role in the topological phase transitions of the bilayer interacting system. For the AA-stacking case, the system realizes quantum spin Hall insulator in the presence (absence) of time-reversal symmetry and magnetically ordered insulator. For the Bernal-stacking case, however, additional phases such as charge-ordered normal insulator or Chern insulator with both charge and magnetic order can be stabilized. Based on our analysis, we discuss the scope of experimental realization in bilayers of transition metal chalcogenides.