Long Short Term Memory Networks for Lexical Normalization of Tweets.

Lexical normalization is converting a non-standard text into a standard text that is more readable and universal. Data obtained from social media sites and tweets often contain much noise and use non-canonical sentence structures such as non-standard abbreviations, skipping of words, spelling errors, etc. Hence such data needs to be appropriately processed before it can be used. The processing can be done by lexical normalization, which reduces randomness and converts the sentence structure to a predefined standard. Hence lexical normalization can help in improving the performance of systems that use user-generated text as inputs. There are several ways to perform lexical normalization, such as dictionary lookups, most frequent replacements, etc. However, We aim to explore the domain of deep learning to find approaches that can be used to normalize texts lexically.