Combining multiple decisions based on dependence relationship

Abstract

In order to overcome difficulties in improving classification performance using only a single classifier, the idea of combining multiple classifiers, as an alternative, has emerged from an assumption that two heads are better than one. The main task of combining multiple classifiers in parallel is how to combine their decisions (i.e., classification results). Although many decision combination methods have been proposed for combining multiple classifiers, most of them have not focused on dependencies among classifiers. They mainly combined multiple decisions on the basis of an independence assumption. Therefore, the performance of combining multiple classifiers tends to be degraded and biased, in case of adding highly dependent classifiers. Huang and Suen proposed Behavior-Knowledge Space (BKS) method, as an advanced study, which would no longer require the independence assumption. However, it is well known that for an application of the BKS method to the combination of K classifiers, storing and estimating a (K+1) st-order probability distribution composed of a decision variable and K decisions is exponentially complex and is unmanageable in theoretical analysis even for small K. Therefore, an approximation scheme is needed. To overcome such weaknesses and obtain robust performance, it is desirable that combining multiple classifiers be performed in a probabilistic way based on the product approximation of the (K+1)st-order probability distribution, without the independence assumption. Chow and Liu as well as Lewis proposed the approximation scheme of a high order probability distribution with a product of second-order distributions considering first-order tree dependency. In their approximation scheme, a dependency provides a theoretical basis on the identification of an optimal approximation of the high order probability distribution, using the measure of closeness proposed by Lewis. However, often we face cases in which a decision is based more than two ot...