Software Support for Coherent Prototyping of 3D Gesture Interactions

When prototyping applications that include touchless 3D gesture interaction three design matters must be taken into consideration: the gestures the user must execute, the visual representation and the dialog flow. Ideally, these aspects should be considered in parallel, to achieve a coherent design process, and avoid ineffective extra effort stemming from coordination between them. A flexible changeover of perspectives among the separate matters is needed. This paper proposes a software environment that enables the desired coherent rapid prototyping of applications with 3D gesture interactions. Its core consists of two types of mapping and a so-called co-simulation functionality. The environment facilitates combining existing software tools from industry and literature to cover the three design matters, i.e. specification and simulation of UI prototypes, gestures, and dialog models. It assists developers at design time in the specification of gestures and in binding them to UI prototypes as well as to statecharts used for defining dialog models. Relevant coherency information is used to offer the option to evaluate gestures at runtime in the context of UI prototype and dialog model. The co-simulator manages the synchronized simulation of all relevant artefacts once a gesture event occurs. Therefore, it enables quickly building prototypes that go beyond the capabilities of the individual tools. This paper describes the usage of the proposed environment in form of a case study with several software tools, each covering one of the three design aspects. It also shows its general applicability, meaning that it can be used with other tools too.