Microscopic origin of the spin-reorientation transition in the kagome topological magnet TbMn6Sn6

TbMn6Sn6 is a correlated topological magnet with a Mn-based kagome lattice, in which a Chern gap opens at the Dirac point at low temperatures. The magnetic moment direction of the ferrimagnetic order changes from in the kagome plane to out-of-plane upon cooling, which is essential for generating the Chern gap, but the underlying mechanism for the spin reorientation remains elusive. Here, we investigate the spin-reorientation transition in TbMn6Sn6 using neutron scattering. We provide direct evidence for the spin-reorientation transition and unveil the coexistence of two Tb modes at 200 K. To account for these results, we put forward a model based on SU(N) spin-wave theory, in which there is a temperature evolution of the ground state Tb 4 f orbitals, driven by the crystalline electric field, single-ion anisotropy, and exchange interactions between Tb and Mn ions. Our findings shed light on the complex magnetism of TbMn6Sn6, despite its relatively simple ground state magnetic structure, and provide insights into the mechanisms for tuning magnetic topological materials.