Human and Large Language Model Intent Detection in Image-Based Self-Expression of People with Intellectual Disability

Non-verbal communication is essential for the social inclusion of individuals with an intellectual disability, affecting interactions with others as well as technological systems. This study focuses on non-symbolic communication of people with intellectual disability through generic images without specific or detailed subject matter. A key challenge in this medium is discerning the underlying intentions behind images selected as visual prompts for conversation. Through interviews with people with intellectual disability, we collected a dataset of images and their associated communication intentions, as well as the interpreted intention from potential audiences including humans and systems. Notably, we employed GPT-4, a Large Language Model (LLM), to decipher the images and provide insights from the Conversational Systems’ (CSs) perspective. Adopting a user-centered qualitative approach, we analyzed this data to understand the nuances of image-based self-expression and identify areas of ambiguity. Additionally, we present an analysis of comprehension levels and challenges for intent detection faced by humans and systems. Our findings suggest that generic images offer a rich medium for individuals to share personal interests and unique experiences, enriching communication. Within this framework, we identified that situational, personal, and historical contexts facilitate understanding intents. In a comparative analysis of human and systems viewpoints, we found that LLMs have encouraging capabilities for detecting various aspects of ambiguity in predicting user intentions. Based on this, we offer design strategies for intent clarification and crafting more inclusive multimodal conversational tools for individuals with intellectual disability. Findings can be extrapolated to enhance image-based information retrieval and recommendation systems.