LiveLocalizer: Augmenting Mobile Text-to-Speech with Microphone Arrays, Optimized Localization and Beamforming

Speech-to-text capabilities on mobile devices have proven helpful for language translation, note-taking, hearing and speech accessibility, and meeting transcripts. However, their usefulness is constrained by being unable to distinguish between multiple speakers, track which direction speech is coming from, and provide acceptable performance in noisy environments. This work introduces efficient real-time audio localization and adaptive beamforming algorithms on custom sound perception hardware running on a low-power microcontroller and four integrated microphones. A prototype is implemented in a phone case form factor and is plug-and-play with modern smartphones. We characterize the performance in technical evaluations of localization, beamforming, and diarization. We demonstrate how the phone case extends existing smartphones with speaker diarization in a speech-to-text app, sound direction visualization, and sound enhancement through beamforming. In the future, we hope our approach will inspire the widespread adoption of advanced microphone arrays that natively unlock the potential of spatial sound processing and perception in mobile and wearable devices.