Faded neural projection from the posterior bed nucleus of the stria terminalis to the lateral habenula contributes to social signaling deficit in male BTBR mice as a mouse model of autism

BTBR T+ Itpr3tf/J (BTBR) mice display several behavioral characteristics, including social deficits resembling the core symptoms of human autism. Atypical social behaviors include sequential processes of assembled cognitive-behavior components, such as recognition, investigatory assessment, and signaling response. This study aimed to elucidate the neural circuits responsible for the regulation of the social signaling response, as shown by scent marking behavior in male mice. We first assessed the recognition and investigatory patterns of male BTBR mice compared to those of C57BL/6 J (B6) mice. Next, we examined their scent-marking behavior as innate social signaling responses adjusted to a confronted feature of social stimuli and situations, along with the expression of c-Fos as a marker of neuronal activity in selected brain areas involved in the regulation of social behavior. The function of the targeted brain area was confirmed by chemogenetic manipulation. We also examined the social peptides, oxytocin and vasopressin neurons of the major brain regions that are associated with the regulation of social behavior. Our data indicate that male BTBR mice are less responsive to the presentation of social stimuli and the expression of social signaling responses, which is paralleled by blunted c-Fos responsivity and vasopressin neurons morphological changes in selected brain areas, including the posterior bed nucleus of the stria terminalis (pBnST) and lateral habenula (LHb) in BTBR mice. Further investigation of LHb function revealed that chemogenetic inhibition and activation of LHb activity can induce a change in scent marking responses in both B6 and BTBR mice. Our elucidation of the downstream LHb circuits controlling scent marking behavior indicates intact function in BTBR mice. The altered morphological characteristics of oxytocin neurons in the paraventricular nucleus of the hypothalamus and vasopressin-positive neurons and axonal projections in the pBnST and LHb appear to underlie the dysfunction of scent marking responses in BTBR mice.