Evaluating and Mitigating Unfairness in Multimodal Remote Mental Health Assessments

Research on automated mental health assessment tools has been growing in recent years, often aiming to address the subjectivity and bias that existed in the current clinical practice of the psychiatric evaluation process. Despite the substantial health and economic ramifications, the potential unfairness of those automated tools was understudied and required more attention. In this work, we systematically evaluated the fairness level in a multimodal remote mental health dataset and an assessment system, where we compared the fairness level in race, gender, education level, and age. Demographic parity ratio (DPR) and equalized odds ratio (EOR) of classifiers using different modalities were compared, along with the F1 scores in different demographic groups. Post-training classifier threshold optimization was employed to mitigate the unfairness. No statistically significant unfairness was found in the composition of the dataset. Varying degrees of unfairness were identified among modalities, with no single modality consistently demonstrating better fairness across all demographic variables. Post-training mitigation effectively improved both DPR and EOR metrics at the expense of a decrease in F1 scores. Addressing and mitigating unfairness in these automated tools are essential steps in fostering trust among clinicians, gaining deeper insights into their use cases, and facilitating their appropriate utilization. Author summary In this work, we systematically explored and discussed the unfairness reporting and mitigation of automated mental health assessment tools. These tools are becoming increasingly important in mental health practice, especially with the rise of telehealth services and large language model applications. However, they often carry inherent biases. Without proper assessment and mitigation, they potentially lead to unfair treatment of certain demographic groups and significant harm. Proper unfairness reporting and mitigation of these tools is the first step to building trust among clinicians and patients and ensuring appropriate application. Using our previously developed multimodal mental health assessment system, we evaluated the unfairness level of using various types of features of the subjects for mental health assessment, including facial expressions, acoustic features of the voice, emotions expressed through language, general language representations generated by large language models, and cardiovascular patterns detected from the face. We analyzed the system’s fairness across different demographics: race, gender, education level, and age. We found no single modality consistently fair across all demographics. While unfairness mitigation methods improved the fairness level, we found a trade-off between the performance and the fairness level, calling for broader moral discussion and investigation on the topic. ### Competing Interest Statement The authors have declared no competing interest. ### Funding Statement Yes ### Author Declarations I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained. Not Applicable The details of the IRB/oversight body that provided approval or exemption for the research described are given below: The Emory University Institutional Review Board and the Grady Research Oversight Committee granted approval for this study (IRB# 00105142). I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals. Not Applicable I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance). Not Applicable I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable. Not Applicable Data cannot be shared publicly since it contains personal identifiable information (video recordings of participants’ faces and audio recordings of speech). Although participants agreed to be recorded for purposes of the work, there are restrictions in place for the release of PII from Emory IRB. Additionally, due to the sensitive nature of the data that we collect, Certificates of Confidentiality are in place to prohibit disclosure of identifying information. Data are available from the Emory department of biomedical informatics (please contact the authors or bmi@emory.edu) for researchers who meet the criteria for access to confidential data.