Scalable algorithms for Bayesian pseudocoresets

Abstract

This paper explores Bayesian pseudocoreset construction, a method for compressing datasets to enable practical Bayesian learning. Bayesian pseudocoresets refer to very small synthetic datasets that mimic the posterior distribution of a model trained on the entire dataset. They can be efficiently applied in Bayesian methodologies where time complexity increases with dataset size. This paper proposes a scalable Bayesian pseudocoreset synthesis algorithm for image data. Firstly, to scale efficiently, we approximate the posterior distribution of both the full dataset and Bayesian pseudocoreset and discuss various divergence measures between these distributions. Additionally, we introduce a novel Bayesian pseudocoreset construction algorithm using the forward KL divergence, more suitable for Bayesian model averaging. Secondly, for scaling with much larger modern deep neural networks, we propose the function space Bayesian pseudocoreset. It models the posterior distribution in the function space rather than the parameter space to accommodate the increased model size. Lastly, we assess the applicability of the synthesized Bayesian pseudocoreset in the contexts of transfer learning and continual learning.