Data association and feature selection with Gaussian processes

Abstract

Gaussian process (GP) has been widely used in supervised learning as a prior process to directly estimate a hidden and complicated relation between input and output. Due to this ability of GP, it is recently applied in several areas of data analysis and machine learning, including data association and feature selection. In this proposal we propose new methods with GPs for data association and feature selection. As the goal of data association is separating observations from different sources, we propose a Bayesian approach based on a mixture of GPs having two key components, the assignment probabilities and the GPs. In the proposed approach, the two key components are simultaneously updated according to observations through an efficient Expectation-Maximization (EM) algorithm that we newly develop. The proposed approach is thus more adaptive to the observations than the existing GP based approaches. We also provide a theoretical analysis to show the effectiveness of the Bayesian update in the proposed approach. Next, we examine a recently proposed feature selection method that measures the feature relevance by estimating dissimilarity between the predictive distributions of GP at a training sample and its perturbed sample by a small amount. However, this existing method with GP suffers from the scalability problem and hence needs refinement for its applicability to large data sets. Moreover, it uses the Kullback-Leibler (KL) divergence in sensitivity analysis for feature selection, but we theoretically show that the KL divergence under-estimates the relevance of important features in some cases of classification. Hence we propose a new method with GP to remedy such limitations of the existing method for better feature selection. Throughout experiments with synthetic and real data sets we show that the proposed methods outperform the existing methods.