The Expert of Oz: A Two-sided Study Paradigm for Intelligent Systems

We present the paradigm of Expert of Oz studies for the formative evaluation of hypothetical AI systems. These studies follow the principle of Wizard of Oz studies but use a human expert for simulating the AI. This allows the experimenter to not only investigate the user's behavior with and reaction to such a system but also to observe and analyze the performance of the expert in the situation at hand. As a consequence, we can learn both about interaction with the hypothetical AI system and about the required AI functionality at the same time and in realistic interaction scenarios.