Keyword Search For Sign Language

Keyword search is the search for a written query in an archive, which is often assumed to be a collection from a spoken language. Yet, the main languages of the Deaf, i.e. sign languages, are mostly neglected in this definition due to being visual languages. In this paper, we propose a keyword search (KWS) system for a sign language. In this technique, we first extract body and hand joints from the frames of a sign language sentence and represent it with a unified spatio-temporal graph of skeleton joints. We then train our graph-convolutional-network encoder, query embedding, and selection mechanism together in a weakly supervised, end-to-end fashion. Experimental results are reported on RWTH-PHOENIX-Weather 2014T dataset.