A Hubel Weisel model for hierarchical representation of concepts in textual documents

A Hubel Weisel model for hierarchical representation of concepts in textual documents Kiruthika Ramanathan (kiruthika_r@dsi.a-star.edu.sg), Shi Luping (shi_luping@a-star.edu.sg), Chong Tow Chong (chong_tow_chong@dsi.a-star.edu.sg) Data Storage Institute, (A*STAR) Agency for Science, Technology and Research, DSI Building, 5 Engineering Drive 1, Singapore 117608 the existing memory, such that the new memory can be acquired without damage to the old ones. Abstract Hubel Weisel models of the cortex describe visual processing as a hierarchy of increasingly sophisticated representations. While several models exist for image processing, little work has been done with Hubel Weisel models out of the domain of object recognition. In this paper, we describe how such models can be extended to the representation of concepts, resulting in a model that shares several properties with the PDP model of semantic cognition. The model that we propose is also capable of incremental learning, in which the knowledge is stored in the strength of the neuron connections. Degradation of old knowledge occurs as new knowledge is introduced to the system in a fashion that simulates decay theory in short term memory. The simulation model therefore captures several properties of cognitive conceptual memory, including generalization patterns, the role of rehearsal and, hierarchical representation. Related work Introduction There exist several bottom-up approaches to hierarchical models of object recognition that are based on the visual cortex. They make use of Mountcastle’s (1978) theory of uniformity and hierarchy in the cortical column and the model of simple to complex cells of Hubel and Weisel (1965), modeling how simple cells from neighboring receptive fields feed into the same complex cell, meaning that the complex cell has phase invariant response. In this paper, we consider the following question. If the structure of the cortical column is uniform and hierarchical in nature and if the model of simple to complex cells can be used to model the visual cortex as discussed in prior works, then can such a model also be used to represent other modalities of information such as the concepts derived from text? We are therefore aiming to design a bottom up hierarchical memory for the representation of concepts, much the same way as it is designed for the representation of images. In this paper, we will define a concept as being a keyword in a document. To deal with the dynamic nature of concept inputs, we look at incremental learning of concepts from two aspects relevant to concept representation from text – (a) with respect to new incoming features and (b) training of hierarchies. To perform this, we apply a set of geometric approximations to the incremental inputs and Mountcastle (1978) showed that parts of the cortical system are organized in a hierarchy and that some regions are hierarchically above others. In general, neurons in the higher levels of the visual cortex represent more complex features with neurons in the IT representing objects or object parts (Hubel and Weisel, 1965). Hubel Weisel models have therefore been developed for object recognition (Cadieu et al., 2007; Fukushima, 2003) proposing a hierarchy of feature extracting simple (S) and complex (C) cells that allow for positional invariance. The combination of S-cells and C-cells, whose signals propagate up the hierarchy allows for scale and position invariant object recognition. The idea of feature based concept acquisition has been well studied in psychological literature. Sloutsky (2003) discusses how children group concepts based on, not just one, but multiple similarities, which tap the fact that those basic level categories have correlated structures (or features). This correlation of features is also discussed in McClelland and Rogers (2003) who argue that information should be stored at the individual concept level rather than at the super ordinate category level allowing properties to be shared by many items. Our model is related to Hubel Weisel approaches in that it implements a hierarchical modular architecture for bottom-up propagation of conceptual information. To our knowledge, however, this is the first implementation of a Hubel Weisel approach to non- natural medium such as text, and has attempted to model hierarchical representation of keywords to form concepts. System architecture The system that we describe here is organized in a bottom up hierarchy. This means that the component features are represented before the representation of concept objects. Our learning algorithm exploits the property of this hierarchical structure. Each level in the