"Don't Forget to Put the Milk Back!" Dataset for Enabling Embodied Agents to Detect Anomalous Situations

Home robots intend to make their users lives easier. Our work assists in thisgoal by enabling robots to inform their users of dangerous or unsanitaryanomalies in their home. Some examples of these anomalies include the userleaving their milk out, forgetting to turn off the stove, or leaving poisonaccessible to children. To move towards enabling home robots with theseabilities, we have created a new dataset, which we call SafetyDetect. TheSafetyDetect dataset consists of 1000 anomalous home scenes, each of whichcontains unsafe or unsanitary situations for an agent to detect. Our approachutilizes large language models (LLMs) alongside both a graph representation ofthe scene and the relationships between the objects in the scene. Our keyinsight is that this connected scene graph and the object relationships itencodes enables the LLM to better reason about the scene – especially as itrelates to detecting dangerous or unsanitary situations. Our most promisingapproach utilizes GPT-4 and pursues a categorization technique where objectrelations from the scene graph are classified as normal, dangerous, unsanitary,or dangerous for children. This method is able to correctly identify over 90of anomalous scenarios in the SafetyDetect Dataset. Additionally, we conductreal world experiments on a ClearPath TurtleBot where we generate a scene graphfrom visuals of the real world scene, and run our approach with nomodification. This setup resulted in little performance loss. The SafetyDetectDataset and code will be released to the public upon this papers publication.