Permutation Decision Trees
arXiv (Cornell University)(2023)
Abstract
Decision Tree is a well understood Machine Learning model that is based on
minimizing impurities in the internal nodes. The most common impurity measures
are Shannon entropy and Gini impurity. These impurity measures are insensitive
to the order of training data and hence the final tree obtained is invariant to
any permutation of the data. This is a limitation in terms of modeling when
there are temporal order dependencies between data instances. In this research,
we propose the adoption of Effort-To-Compress (ETC) - a complexity measure, for
the first time, as an alternative impurity measure. Unlike Shannon entropy and
Gini impurity, structural impurity based on ETC is able to capture order
dependencies in the data, thus obtaining potentially different decision trees
for different permutations of the same data instances, a concept we term as
Permutation Decision Trees (PDT). We then introduce the notion of Permutation
Bagging achieved using permutation decision trees without the need for random
feature selection and sub-sampling. We conduct a performance comparison between
Permutation Decision Trees and classical decision trees across various
real-world datasets, including Appendicitis, Breast Cancer Wisconsin, Diabetes
Pima Indian, Ionosphere, Iris, Sonar, and Wine. Our findings reveal that PDT
demonstrates comparable performance to classical decision trees across most
datasets. Remarkably, in certain instances, PDT even slightly surpasses the
performance of classical decision trees. In comparing Permutation Bagging with
Random Forest, we attain comparable performance to Random Forest models
consisting of 50 to 1000 trees, using merely 21 trees. This highlights the
efficiency and effectiveness of Permutation Bagging in achieving comparable
performance outcomes with significantly fewer trees.
MoreTranslated text
Key words
trees,decision
AI Read Science
Must-Reading Tree
Example
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined