EyeClick: A Robust Two-Step Eye-Hand Interaction for Text Entry in Augmented Reality Glasses

Efficient text entry is a crucial aspect of the user experience for augmented reality (AR) head-mounted displays (HMD). Eye-tracking for virtual keyboard interaction is a popular choice for AR text entry, as it is intuitive, privacy-preserving, and socially acceptable. However, as AR HMDs move toward more consumer-friendly form factors and price points, technical constraints necessitate trade-offs that result in a limited field of view (FoV) and reduced eye-tracking accuracy. To address these challenges, we develop EyeClick, a novel two-step eye-hand interaction method utilizing a modified QWERTY keyboard for AR HMDs. To type, users first employ gaze to select a large area containing multiple characters, then press a button on a handheld controller to choose a single character from the selected area. We demonstrated competitive speed and reduced corrected error rate (CER) with two variants of the EyeClick design: ColType (7.30 WPM) and RowType (9.41 WPM). We also integrated ChatGPT with the EyeClick keyboard to build an interactive information retrieval AR application. We demonstrate that the EyeClick keyboard can help users efficiently extract information about surroundings by typing to ChatGPT in mixed reality.