pgeon applied to Overcooked-AI to explain agents' behaviour.

Policy Graphs (PGs) are a method for representing the behaviour of opaque agents by observing them in the environment and producing graphs where the state and action spaces are discretised into predicates. We present pgeon, a Python library that demonstrates the effectiveness of PGs in providing explanations for the behaviour of agents and we showcase it by applying it to a multi-agent cooperative environment: Overcooked-AI. This library illustrates how PGs can create transparent and explainable surrogate agents that closely mimic the behavior of the original agents. These features can help improving trust in environments where humans and AI systems collaborate by improving the explainability of all agents, even opaque or human.