Research on Index Mechanism of HBase Based on Coprocessor for Sensor Data

In order to provide effective management of big data, almost two hundred different NoSQL stores have been developed, among which HBase is one of the best known. When performing data queries, the native HBase supports primary key indexes well. For non-primary key data, only the full table scan can be used, which greatly reduces the multi-condition query speed of HBase. Some additional indexing techniques have been presented to support querying on non-key indexing for HBase. For large-scale sensor data, this research field still requires effective design paradigm, in-depth experimentations and practical implementations. Therefore, we propose and implement a secondary memory index mechanism of HBase based on coprocessor for sensor data. The index mechanism can be automatically updated according to the change of the HBase table. Meanwhile, the index mechanism is persisted and maintained in memory, which can greatly improve the retrieval speed of the index data. By using real sensor datasets with different distribution patterns, experimental results show that the condition retrieval speed of the mechanism is greatly improved compared with the original HBase. In addition, compared with the secondary index mechanism based on Solr and Hibase, the performance of our proposed solution is also improved.