Hand-and-finger-awareness for mobile touch Interaction using deep learning

Mobile devices such as smartphones and tablets have replaced desktop computers for a wide range of everyday tasks. Virtually every smartphone incorporates a touchscreen which enables an intuitive interaction through a combination of input and output in a single interface. Due to the success of touch input, a wide range of applications became available for mobile devices which were previously exclusive to desktop computers. This transition increased the mobility of computing devices and enables users to access important applications even while on the move. Despite the success of touchscreens, traditional input devices such as keyboard and mouse are still superior due to their rich input capabilities. For instance, computer mice offer multiple buttons for different functions at the same cursor position while hardware keyboards provide modifier keys which augment the functionality of every other key. In contrast, touch input is limited to the two-dimensional location of touches sensed on the display. The limited input capabilities slow down the interaction and pose a number of challenges which affect the usability. Among others, shortcuts can merely be provided which affects experienced users and contradicts Shneiderman’s golden rules for interface design. Moreover, the use of mostly one finger for input slows down the interaction while further challenges such as the fat-finger problem and limited reachability add additional inconveniences. Although the input capabilities are sufficient for simple applications, more complex everyday tasks which require intensive input, such