Learning over categorical data using counting features: with an application on click-through rate estimation

Input data for many machine learning applications are often categorical and contain multiple fields. A common feature representation for such categorical data is one-hot encoding, which expresses data instances as high-dimensional sparse binary vectors. Given this encoding machine learning models, such as logistic regression or boosted trees, are trained. However, the following problems occur when dealing with large-scale data sets: (i) The binary feature space is sparse yet extremely large, which can require a large amount of computational resources. (ii) Models based on such a feature representation will typically need to be re-trained in order to keep up-to-date with any changes in the data distribution. (iii) The one-hot feature representation provides little generalisation ability. In this paper, we propose counting features, a novel statistics-based feature engineering paradigm, to address the above problems. Mathematically, we show a deterministic relationship between the optimal regression parameters of counting features and one-hot binary features. Then, in the context of click-through rate estimation in online advertising, we demonstrate that counting features indeed bring better generalisation ability. Our experiments on real-world large-scale datasets demonstrate that, despite their compressed nature, the proposed counting feature engineering outperforms the one-hot binary encoded features in various cases such as cold start training and cross-campaigns training.