Large-margin and dropout learning of sum-product networks

Abstract

This paper investigates the generalization property of a sum-product network (SPN) in which the parameters are learned based on large-margin criterion and dropout. The learning criterion minimizes an objective function defined as a sum of an $L_{2}$ regularizer and a smooth hinge function of the margin that is defined as the smallest difference in unnormalized conditional probability between the true label and its closest competitor. The partial derivatives with respect to all the parameters of the SPN can be computed with backpropagation. To further improve the generalization of the SPN, the sum nodes (or max nodes in max-product network) are dropped out with Bernoulli distribution of a half during learning: dropout reduces overfitting by model-averaging. The large-margin based learning outperforms previously proposed discriminative learning of SPNs on CIFAR-10 and STL-10 for various training data size. The performance gain with large-margin learning increases with decrease in the number of training data. With dropout, the performance of large-margin is further improved for the two benchmark datasets.