Real Time Sound Processing on Android

The introduction of the Internet of Things (IoT) and the ubiquity of mobile devices has generated interest in leveraging smartphones for deploying and running sophisticated mobile sound applications. Modern smartphones have good connectivity, a significant amount of processing, and are always with us, making them an ideal candidate for envisioned applications. Many such applications have stringent latency requirements and in some cases, special I/O capabilities. One example is a class of real-time audio applications that require the ability to overlay sounds of different frequencies for joint playback. Currently, Android is unable to provide such tight timing requirement due to the design decisions for the underlying platform. In this paper, we present a new architecture for scheduling and managing sound playback that supports overlays. We show that our prototype has competitive performance and significantly reduces latency spikes compared to stock Android. We evaluate our system on both micro benchmarks as well as sound applications including a wind farm monitoring application that leverages sound to detect micro fractures and a multiphase surround sound playback application that leverages sound overlays to coordinate between multiple devices.