Towards designing human-centered artificial intelligence for computer vision tasks: A focus group study on preventing medication dispensing errors (Preprint)

BACKGROUND Medication errors including dispensing errors represent a substantial health risk globally with significant implications in terms of morbidity, mortality, and financial costs. Although pharmacists utilize methods like barcode scanning and double-checking for dispensing verification, these measures exhibit limitations. In this light, the application of artificial intelligence (AI) in pharmacy verification emerges as a potential solution, offering precision, rapid data analysis, and the ability to recognize medications through computer vision. Yet, for AI to be embraced, it must be designed keeping the end-user in mind, fostering trust, clear communication, and seamless collaboration between AI and healthcare professionals. OBJECTIVE This study investigated the development of a human-centered AI system to enhance pharmacists' medication dispensing verification process. We aim to identify pharmacists' challenges and explore AI's potential for fostering collaboration and trust for medication dispensing verification. METHODS A multidisciplinary research team engaged pharmacists in a three-stage process to develop a human-centered AI system for medication dispensing verification. To design the AI model, we employed a Bayesian neural network that predicts the dispensed pills' National Drug Code (NDC). Discussion scripts regarding how to design the system and feedback in focus groups were collected via audio recordings and professionally transcribed, followed by a content analysis guided by the Systems Engineering Initiative for Patient Safety (SEIPS) and Human-Machine Teaming (HMT) theoretical frameworks. RESULTS Eight pharmacists participated in 3 rounds of focus groups to identify current challenges in medication dispensing verification, brainstorm solutions, and provide feedback on our AI prototype. Participants considered several teaming scenarios, generally favoring a hybrid teaming model where the AI assists in the verification process and a human steps in based on medication risk level and the AI's confidence level. Pharmacists highlighted the need for improving the interpretability of AI systems, such as adding stepwise checkmarks, probability scores, and confusing reference pills (the second-highest AI-predicted NDC). We visualized the human-centered AI prototype based on the pharmacists’ insights and feedback. CONCLUSIONS In partnership with pharmacists, we developed a human-centered AI prototype designed to enhance AI interpretability and foster trust. This initiative emphasized human-machine collaboration and positioned AI as an augmentative tool rather than a replacement. Our study highlights the importance of human-centered AI in overcoming medication dispensing verification challenges, emphasizing interpretability, confidence visualization, and harmonious human-machine teaming styles in pharmacy practice. Future research should focus on validating the system's performance in real-world scenarios.