When ELIZA Meets Therapists: A Turing Test for the Heart and Mind

“Can machines be therapists?” is a question receiving increased attention given the relative ease of working with generative artificial intelligence. Although recent (and decades-old) research has found that humans struggle to tell the difference between responses from machines and humans1,2, recent findings suggest that artificial intelligence can write empathically and the generated content is rated highly by therapists and outperforms professionals.3,4,5,6,7,8 It is uncertain if in a preregistered competition where therapists and ChatGPT respond to therapeutic vignettes about couple therapy whether a) a panel of participants (N = 830) can tell which responses are ChatGPT-generated and which are written by therapists, b) the generated responses or the therapist-written responses fall more in line with key therapy principles, and c) linguistic differences between conditions are present. We show with a large sample (N = 830) that a) participants can rarely tell a difference between responses written by ChatGPT and responses written by a therapist, b) the responses written by ChatGPT are generally rated higher in key psychotherapy principles, and c) the language patterns between ChatGPT and therapists are different. Using different measures, we then confirm that responses written by ChatGPT were rated higher than the therapist's responses suggesting these differences may be explained by part-of-speech and response sentiment. This may be an early indication that ChatGPT has the potential to improve psychotherapeutic processes. We anticipate that this work may lead to the development of different methods of testing and creating psychotherapeutic interventions. Further, we discuss limitations (including the lack of the therapeutic context), and how continued research in this area may lead to improved efficacy of psychotherapeutic interventions allowing such interventions to be placed in the hands of individuals who need them the most.