MechTac: A Multifunctional Tendon-Linked Optical Tactile Sensor for In/Out-the-Field-of-View Perception with Deep Learning

Tactile sensors can be used for motion detection and object perception in robot manipulation. The contact detection within the camera's visual inspection area has been well-developed, but perception outside the field of view of the camera is overlooked. In this paper, we present a new tendon-linked tactile sensor, MechTac, to achieve perceptions inside and outside the field of view. The MechTac is an evolution of the typical TacTip sensor with the following two advantages. 1) The ability to provide perception outside the field of view. This is achieved by using a network of braided tendons to transfer deformation from the blind perception regions (TacSide) to the visual areas (TacTip). 2) The tactility of the TacSide and TacTip is reflected by the movements of multiple papillae pins and visible markers on the pin tips on the inner surface of the TacTip. The pins and markers are differentially sensitive to various touch features, which is similar to the differentiated perceptual ability of humans. TacTip is more sensitive to small touches, corresponding to the fingertip, while the TacSide is less sensitive but has a larger perceptual area, corresponding to the middle part of the finger. Moreover, we propose a new deep learning method to decompose the mixed information affected by the TacSide and the TacTip. A modified DenseNet121 was specifically designed for object perception at the TacTip and localization at the TacSide. The experimental results show that prediction accuracy reaches about 98% for object perception or localization and over 99% for the case requiring two functions.