Label Input Thread Audio Requesting Thread Label ringbuffer Reactive Modification Thread Prosody Model Statistical Model Audio ringbuffer MAGE Parse Label Update PDFs Update Filter Update Audio

Speech production is a complex phenomenon with many parameters. It is very difficult for one performer to control all aspects of a synthesizer that models this phenomenon. We designed and developed a distributed, multi-user system to tackle this difficulty, where users control different aspects of the synthesizer simultaneously and interactively; treating the complex production process as a social game. HMM-based synthesizers provide flexibility at a high level of naturalness, thus we chose HTS as our synthesizer. However, HTS needs severe architectural modifications to be used reactively, and a major achievement of this work was creating MAGE/pHTS, a library for performative HMM-based speech and singing synthesis. The resulting system provides interactive controls for phonetic content and context, as well as prosody using the previously existing HandSketch controller.