Towards Semantic Action Analysis via Emergent Language

Recent work on unsupervised learning has explored the feasibility of semantic analysis and interpretation via Emergent Language (EL) models. As EL requires some form of numerical embedding, it remains unclear which type is required in order for the EL to properly capture certain semantic concepts associated with a given task. In this paper, we compare different approaches that can be used to generate such embeddings: unsupervised and supervised. We start by producing a large dataset using a single-agent simulation environment. In these experiments, a purpose-driven agent attempts to accomplish a number of tasks. These tasks are performed in a synthetic cityscape environment, which includes houses, banks, theaters, and restaurants. Given such experiences, specification of the associated goal structure constitutes a narrative. We investigate the feasibility of producing an EL from raw pixel data with the hope that resulting descriptions can be used to infer the underlying narrative structure. Our initial experiments show that a supervised learning approach yields embeddings and EL descriptions that capture narrative structure. Alternatively, an unsupervised learning approach results in greater ambiguity with respect to the narrative.