Absence of magnetic order and emergence of unconventional fluctuations in the Jeff=21 triangular-lattice antiferromagnet YbBO3

We present the ground state properties of a new quantum antiferromagnet, YbBO3, in which the isotropic Yb3+ triangular layers are separated by a nonmagnetic layer of partially occupied B and O(2) sites. The magnetization and heat capacity data establish a spin-orbit entangled effective spin Jeff = 1/2 state of Yb3+ ions at low temperatures, interacting antiferromagnetically with an intralayer coupling J/kB similar or equal to 0.53 K. The absence of oscillations and a 1/3 tail in the zero-field muon asymmetries rules out the onset of magnetic long-range order as well as spin freezing down to 20 mK. An anomalous broad maximum in the temperature-dependent heat capacity with an unusually reduced value and a broad anomaly in the zero-field muon depolarization rate centered at T* similar or equal to 0.7J/kB provide compelling evidence for a wide fluctuating regime (0.2 less than or similar to T/J <= 1.5) with slow relaxation. We infer that the fluctuating regime is a universal feature of highly frustrated triangular-lattice antiferromagnets while the absence of magnetic long-range order is due to perfect two-dimensionality of the spin lattice protected by nonmagnetic site disorder.